Cisco Unified Workforce Optimization Quality Management Troubleshooting Guide 2.7(3) September 2009 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. CCDE, CCENT, Cisco Eos, Cisco HealthPresence, the Cisco logo, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0812R) Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental. Quality Management Troubleshooting Guide © 2008, 2009 Cisco Systems, Inc. All rights reserved. © 2008, 2009 Calabrio, Inc. All rights reserved. Contents Quality Management Troubleshooting Guide 5 ■ Quality Management Documentation 5 ■ Service Names and Executables 6 ■ Registry Entries 7 Site Setup 7 Ident 8 ■ Logs and Debugging 10 Setting the Default Debugger 10 Log Message Formats 12 Configuration Files 13 Enabling Debugging 14 Disabling Debugging 16 ■ ContactDelete Utility 18 Using the Utility 18 ■ Moving the Recording Storage Location 19 ■ Audit Trail Error Messages 23 ■ Troubleshooting Issues 28 Installation Issues 28 QM Configuration Setup Issues 30 Recording Issues 30 QM Administrator Issues 36 QM Desktop Issues 38 Service Issues 39 ■ Troubleshooting Flowcharts 41 The QM Desktop Recording Service is not Recording Audio 41 The QM Desktop Recording Service is not Recording the Screen 48 QM Desktop does not Play Back Recordings 50 The QM Network Recording service is not Recording Client Calls 51 Index 53 Contents Quality Management Troubleshooting Guide Quality Management Documentation The following documents contain additional information about QM 2.7(3): September 2009 ■ Quality Management Installation Guide ■ Quality Management Administrator User Guide ■ Quality Management Desktop User Guide ■ Quality Management Error Code Dictionary ■ Quality Management API Programmers Guide ■ Quality Management 2.7(3) Release Notes 5 Quality Management Troubleshooting Guide Service Names and Executables The following are the QM services as listed in the Windows Services utility. Table 1. 6 QM services and executables Service Name Executable Quality Management CTI Service splkjservice.exe Quality Management DB Cleaner Service splkjservice.exe Quality Management DB Proxy Service splkjservice.exe Quality Management Desktop Recording Service sqmservice.exe Quality Management LDAP Monitor Service LDAPMonSvr.exe Quality Management Mana Service splkjservice.exe Quality Management Monitor Service MonitorServer.exe Quality Management Network Recording Service RecordServer.exe Quality Management Sync Service DirAccessSynSvr.exe Quality Management Tomcat Service tomcat5.exe Quality Management Upload Controller SQMUploadController.exe PROXY Pro Gateway Service PgSvc.exe September 2009 Registry Entries Registry Entries Site Setup HKEY_LOCAL_MACHINE\SOFTWARE\Spanlink\SQM\Site Setup Table 2. Site setup registry entries Value Type Description APP VERSION string Version of the QM software BRAND string Brand of QM software installed 1: Calabrio 0: Cisco CALLCENTERLANG dword Software localization language FirstRun dword 1: QM Configuration Setup has run to completion 0: QM Configuration Setup has not run to completion INSTALL DIRECTORY string Root directory of the QM installation INSTALLDIR string Base directory into which QM software is installed IOR HOSTNAME string Host name or IP address of the QM services LDAP Bind DN string User ID used to log into the LDAP service LDAP Connection Timeout dword Maximum time in seconds before a connection attempt times out LDAP Heartbeat Enabled dword 1: LDAP heartbeats enabled LDAP Heartbeat Retry Time dword Heartbeat time, in milliseconds LDAP Host 1 string Host name or IP address of the LDAP service LDAP Host 2–5 string Host name or IP address of the backup LDAP service(s) LDAP LCC string Default logical contact center LDAP Port 1 dword LDAP service port number September 2009 0: LDAP heartbeats not enabled 7 Quality Management Troubleshooting Guide Table 2. Site setup registry entries — Continued Value Type Description LDAP Port 2–5 dword Backup LDAP service(s) port number(s) LDAP Pwd string LDAP service user password (encrypted) LDAP Recovery Retry Time dword LDAP recovery time, in milliseconds LDAP Request Timeout dword Maximum time in seconds before an LDAP request times out LDAP Root string Root of the LDAP data Monitor Device string The ID of the network adapter used to filter Real-time Transport Protocol (RTP) packets for voice recording Serial Number dword Serial number used to reset LDAPClient if the LDAP host IP changes Ident HKEY_LOCAL_MACHINE\SOFTWARE\Spanlink\SQM\Ident Table 3. 8 Ident registry entries Key Value Type Description Admin (default) string 1 if the services/applications associated are installed; key is absent otherwise Playback (default) string 1 if the services/applications associated are installed; key is absent otherwise Record (default) string 1 if the services/applications associated are installed; key is absent otherwise CTI (default) string 1 if the services/applications associated are installed; key is absent otherwise DB (default) string 1 if the services/applications associated are installed; key is absent otherwise September 2009 Registry Entries Table 3. Ident registry entries — Continued Key Value Type Description LDAP (default) string 1 if the services/applications associated are installed; key is absent otherwise Screen (default) string 1 if the services/applications associated are installed; key is absent otherwise Voice (default) string 1 if the services/applications associated are installed; key is absent otherwise Monitor (default) string 1 if the services/applications associated are installed; key is absent otherwise September 2009 9 Quality Management Troubleshooting Guide Logs and Debugging Applications and services use logging to report status and problems. Each application and service creates two files: ■ Log files (files with the *.log file extension) contain status messages and, if problems occur, warning and other error messages. All messages in log files are identified by an error code. See the Quality Management Error Code Dictionary for more information on error codes. ■ Debugging files (files with the *.dbg file extension) are empty when debugging is not enabled. When debugging is enabled (the default setting), the files contain diagnostic information that can help resolve issues. Log and debugging files are located in the …\Cisco\WFO_QM\log folder on the client or server computer. The default configuration settings limit each log and debugging file to a maximum of 10 MB and 20 rolling files for QM services and 5 MB and 5 rolling files for applications. For example, when a service’s log or debug file reaches 10 MB, it is closed and renamed, and a new file is started. C++ configuration files (files with the *.cfg extension) produce logs using this numbering scheme: <name>0001.log is created and filled. <name>0002.log is created when the first file is full. <name>0001.log is cleared and reused when the second file is full. <name>0002.log is cleared and reused when the first file is full. And so on. Java configuration files (files with the *.properties extension) produce logs using this numbering scheme: <name>.log is always the file currently being filled. <name>.log.1 is the most recent filled file. Debugging logs follow these same numbering schemes, but use the *.dbg file extension. Setting the Default Debugger When Microsoft SQL Server 2005 is installed on the Database Services server, Microsoft Visual Studio is also installed and configured as the default debugger on that server. It is recommended that you reset Dr. Watson for Windows as your default debugger. 10 September 2009 Logs and Debugging To reset Dr. Watson as your default debugger: 1. On the server hosting the QM Database services, click Start > Run. 2. In the Open field, type drwtsn32.exe and click OK. Dr. Watson for Windows starts. 3. Configure the settings to match those shown in Figure 1. ■ Accept the default Log File Path and Crash Dump locations. ■ Set the Crash Dump Type to Full. ■ Clear the Visual Notification check box. 4. Click OK. Figure 1. Dr. Watson configuration settings 5. Click Start > Run. September 2009 11 Quality Management Troubleshooting Guide 6. In the Open field, type drwtsn32.exe -i and click OK. Tthe confirmation dialog is displayed (Figure 2). Figure 2. Dr. Watson confirmation dialog box 7. Click OK to close the dialog box. Dr. Watson is now set as your default debugger. Log Message Formats The following are the formats used by the various log and debug file messages. Each format is followed by an example. C++ and Java *.log file messages <timestamp> <level> <error code> <error text> 2007-02-28 09:29:11.723 INFO ABCD1234 Successfully launched update. C++ *.dbg file messages <timestamp> <level> [<thread ID>] <text> 2007-02-28 14:51:13.723 DEBUG [0xaa8] CSqmcApiBase::_doRecovery: Connected to QM Controller. Java *.dbg file messages <timestamp> <level> [<thread name>|<class>#<method>:<line>] <text> 2007-04-07 15:04:31.954 STACK [Thread-2|Init#run:113] Started. Java (log4j) *.log file messages <timestamp> <level> [<thread name>] [<class>] <text> 2007-04-07 14:54:00,067 INFO [Thread-2] [com.calabrio.morepackages.Init] Started. 12 September 2009 Logs and Debugging Configuration Files Each application and service has an associated configuration file that controls logging and debugging. These files can be edited in a text editor such as Windows Notepad to change the logging and debugging parameters. Configuration files are located in the …\Cisco\WFO_QM\config folder on the client or server computer. CAUTION: Edit configuration files only as described in this section. Improper changes can result in logging and/or program failure, including the possible loss of data. You might want to make a safety backup of any file you edit before you make changes to it. The QM configuration and log files are displayed in Table 4. Table 4. QM configuration and log files Service/Application Location Configuration File Log File Backup and Restore server bars.properties bars.log ContactDelete utility server ContactDelete.properties ContactDelete.log DB Cleaner server dbcleaner.properties dbcleaner.log DB Cleaner server splkSqmDBcleaner.cfg splkSqmDBcleaner.log DB Proxy server dbproxy.properties dbproxy.log DB Proxy server splkSqmDBproxy.cfg splkSqmDBproxy0001.log DB Sync server dbsync.properties dbsync.log File Transfer Servlet (FTS) server log4j-fts.properties* fts.log LDAP Monitor Service server LDAPMonSvr.cfg LDAPMonSvr0001.log LDAP Monitor Service server slapd.cfg sladp0001.log LDAP Monitor Service server slurpd.cfg slurpd0001.log License servlet server licensing.properties licensing.log QM Administrator client admin.properties admin.log QM Configuration Setup server, client postinstall.properties postinstall.log QM Configuration Setup server sitedefaults.properties postinstall.log QM CTI Service server ctiservice.properties ctiservice.log QM CTI Service server splkSqmCTIservice.cfg splkSqmCTIservice0001.log QM Desktop client qmdesktop.properties qmdesktop.log September 2009 13 Quality Management Troubleshooting Guide Table 4. QM configuration and log files — Continued Service/Application Location Configuration File Log File QM Desktop Recording client sqmService.cfg sqmService0001.log QM Mana Service server manaEmergency.properties QM Mana Service server manaservice.properties mana.log QM Mana Service server splkSqmManaService.cfg splkSqmManaService0001.log QM Monitor Service server MonitorServer.cfg MonitorServer0001.log QM Network Recording server RecordServer.cfg RecordServer0001.log QM Reporting server log4j.properties† ReportEngine.log ‡ QM Reporting server log4j_qmr.properties SQMR.log QM Uploader server dbUpload.properties dbUpload.log QM Uploader server SQMUploadController.cfg SQMUploadController0001.log QM Sync Service server DirAccessSynSvr.cfg DirAccessSynSvr0001.log True Update client splkUpdate.cfg splkUpdate0001.log * By default, this configuration file is located in the C:\Program Files\Cisco\WFO_ QM\Tomcat\webapps\fts\WEB-INF\config\ folder. † By default, this configuration file is located in the C:\Program Files\Cisco\WFO_ QM\Tomcat\webapps\CalabrioReportEngine\WEB-INF\classes\ folder. ‡ By default, this configuration file is located in the C:\Program Files\Cisco\WFO_ QM\Tomcat\webapps\QMR\WEB-INF\config\ folder. Enabling Debugging By default, debugging is enabled. When debugging is enabled, keep in mind that the more detail the debugging threshold provides, the slower the performance of your PC and the bigger the size of the debug file. High debugging thresholds may also affect the performance of other applications running on your PC. There are three types of configuration files: ■ C++ files that use the *.cfg extension ■ Java files that use the *.properties extension ■ Java files whose file name begins with “log4j” Each type of file uses a different syntax to enable debugging. The procedures below describe the steps that must be followed for each type of file. IMPORTANT: Disable debugging when it is no longer needed. 14 September 2009 Logs and Debugging The available debugging thresholds are displayed in Table 5. NOTE: Not all thresholds can be used in all configuration files. See the procedures below for which thresholds can be used is particular files. Table 5. Debugging Thresholds Threshold Description Debug Usually sufficient for diagnosing a problem. Will not affect system performance. Call Tracks function entry and exit. Trace Provides a large amount of diagnostic information. May affect system performance. Stack Provides only stacktraces, which give more debugging information when errors and warnings occur. Dump Provides a very large amount of detailed diagnostic information. Likely to affect system performance. Off Turns off debugging. To enable debugging in files with a *.cfg extension: 1. In a text editor, open the desired configuration file. 2. Under the section headed [Debug Log], set the debugging threshold to DEBUG, CALL, TRACE, or DUMP. For example: Threshold=DEBUG The line might already exist or you might have to add a new line. 3. Save the configuration file. The change takes effect immediately. You do not have to restart the application or service. To enable debugging in files with a *.properties extension: 1. In a text editor, open the desired configuration file. 2. Locate the line that starts with: log4j.rootLogger=<threshold>#com.spanlink … and replace <threshold> with DEBUG, TRACE, STACK, or DUMP. 3. Locate the line that starts with: September 2009 15 Quality Management Troubleshooting Guide log4j.appender.DBG.Threshold=<threshold>#com.spanlink … and replace <threshold> with the same value you used in Step 2. 4. Save the configuration file. The change takes effect according to the splk4j.watch.check setting (by default, within 90 seconds). You do not have to restart the application or service. To enable debugging in log4j files: 1. In a text editor, open the desired configuration file. 2. Locate the line that starts with: log4j.rootLogger=<threshold> … and replace <threshold> with DEBUG or TRACE. 3. Save the configuration file. 4. Restart the application or service for the new setting to go into effect. Disabling Debugging It is important to disable debugging when it is no longer needed for diagnostic purposes. Debugging can affect the performance of your PC if it is left enabled. To disable debugging in files with a *.cfg extension: 1. In a text editor, open the desired configuration file. 2. Under the section headed [Debug Log], set the debugging threshold to OFF. For example: Threshold=OFF 3. Save the configuration file. The change takes effect immediately. You do not have to restart the application or service. To disable debugging in files with a *.properties extension: 1. In a text editor, open the desired configuration file. 2. Locate the line that starts with: log4j.rootLogger=<threshold> … and replace <threshold> with STACK. 3. Locate the line that starts with: log4j.appender.DBG.Threshold=<threshold> … 16 September 2009 Logs and Debugging and replace <threshold> with OFF. 4. Save the configuration file. The change takes effect according to the splk4j.watch.check setting (by default, within 90 seconds). You do not have to restart the application or service. To disable debugging in log4j files: 1. In a text editor, open the desired configuration file. 2. Locate the line that starts with: log4j.rootLogger=<threshold> … and replace <threshold> with INFO. 3. Locate the line that starts with: log4j.appender.DBG.Threshold=<threshold>… and replace <threshold> with DEBUG. 4. Save the configuration file. 5. Restart the application or service for the new setting to go into effect. September 2009 17 Quality Management Troubleshooting Guide ContactDelete Utility The ContactDelete utility (ContactDelete.exe) is a command line tool that is installed on the Database Services server in the following location: …\Cisco\WFO_QM\bin\ContactDelete.exe It is used to permanently delete a specified contact from the database. Once this utility is run on a record, no calculations or contact lists will include the deleted contact. Other than the log file, there is no audit trail indicating the record was deleted. The deletion is permanent—the contact cannot be recovered. The utility’s logs are defined in a properties file located at: …\Cisco\WFO_QM\config\ContactDelete.properties The log files are: …\Cisco\WFO_QM\log\ContactDelete.log …\Cisco\WFO_QM\log\ContactDelete.dbg Using the Utility The syntax for this utility is as follows: ContactDelete.exe <options> <uniqueID> where: 18 ■ <uniqueID> - The unique ID of the contact to be deleted. ■ <options> - Optional command parameters. The available command parameters are as follows. ■ -h or --help - Describes utility usage and displays any options. ■ -f or --force - Deletes contact without prompting for deletion confirmation. September 2009 Moving the Recording Storage Location Moving the Recording Storage Location Follow this procedure to change the location where voice and screen recordings are stored. ■ Voice Recordings: You can change the storage location to any local or external folder. It is not necessary that they be stored on the machine hosting the Voice Services. ■ Screen Recordings: You can change the storage location to any local or external folder. It is not necessary that they be stored on the machine hosting the Screen Services. If the Screen Services and Voice Services are on the same server, you can elect to use the same path as is used for the voice recordings. NOTE: QM does not support storing recordings of the same type (voice or screen) in multiple locations. If you set a new storage location, all recordings of the same type must be moved to the new location. The procedure requires some of the QM services and applications to be stopped while the process occurs. However, recording can still take place. IMPORTANT: The File Transfer Servlet that is part of the Voice and Screen services must run as a user (account) with access to whatever location you choose for recordings. To move the voice and/or screen recording storage location: 1. Exit and close QM Administrator. 2. Stop the QM Uploader Controller service. This prevents recordings from being uploaded while the storage location is moved, and provides time to test if recordings can be retrieved from the new location. 3. Back up the QM database and LDAP for disaster recovery purposes. See the section, “Backup and Restore” in the Quality Management Administrator User Guide for the backup procedure. 4. On the QM Services server, navigate to …\Cisco\WFO_QM\bin\ and double-click PostInstall.exe to start the QM Configuration Setup utility. September 2009 19 Quality Management Troubleshooting Guide 5. From the menu, choose Tools > Set Recording Home Directory to display the Recording File Storage Location window (see Figure 3). Figure 3. Recording File Storage Location window 6. Select if you want to store recordings in a local or external storage location, and then enter the desired location in the Host Name/IP Address and Storage Location fields. 20 ■ Local Storage Location. A location that is physically part of the voice or screen server, such as an internal or external hard drive. ■ External Storage Location. A location that is not physically part of the voice or screen server, such as a network drive. September 2009 Moving the Recording Storage Location 7. If you selected an external location, enter the username and password required to access that location. If the user is a domain user, enter the name with the format <doman>\<username>. This user must meet these requirements: ■ The user must be known to the local server (must be a local user or a trusted domain user). ■ If the user is a domain user, the domain specified has to be trusted by the local server. This means the QM Recording Server being configured has to be on a domain that is in (or is trusted by) the domain entered. ■ The user must be able to log on as a service. ■ The user must have read/write access to both the external drive location entered AND the location where QM is installed on the local server. 8. Click OK. 9. Copy the recording files from the audio folder and all subfolders (for voice recordings) or from the video folder and all subfolders (for screen recordings) at the old location (default location is C:\Program Files\Common Files\ QM\recordings\) to the new location. The new location must have the audio or video folder and all its subfolders. For example: Copy: …old location\audio\ …old location\video\ To: …new location\audio\ …new location\video\ 10. Rename the old location audio folder and/or video folder for backup purposes. After you have verified that the files in the new location can be accessed (see the procedures below) and the move was successful, this folder can be deleted. To verify that the moved recordings can be played back: 1. Restart the QM DB Proxy Service and the QM Tomcat service. 2. Start QM Desktop and play back at least one QM recording (if applicable) and one archive recording (if applicable). 3. If the playbacks are successful, restart the QM Upload Controller. September 2009 21 Quality Management Troubleshooting Guide To verify that new recordings are uploaded to the new location: 1. With archiving enabled, make a test call from a phone that is configured to be recorded. 2. In QM Desktop, download that call on demand on the Archive tab and play it. You will be able to do this if the call is uploaded to the correct location. NOTE: If you do not enable archiving, you must wait for the normal uploading cycle to occur before you can review the call on the Recording tab. 22 September 2009 Audit Trail Error Messages Audit Trail Error Messages Table 6 describes the audit trail error messages that can appear in the User Recording Status and System Status reports generated through QM Desktop. Table 6. Audit trail error messages Message Description/Action At least one required component for screen recording is not running. Description: A component required to record the screen is either not running or is not installed. Action: In the Windows Services utility in Control Panel, verify that the PROXY Pro Host Service is installed and running. Conversion from raw to spx failed. Description: Failed to convert the voice file from *.raw to *spx format. Action: QM Desktop Recording service will convert the file later. Description: The extensions from the agent’s phone cannot be found in the inclusion list. No recording will take place. Action: Add the extension(s) you want to record to the inclusion list. Description: The extension number(s) for the logged-in user have changed. Action: None. Failure to update metadata record due to invalid value for this metadata type <key>:<value>:<key>, <invalid value> Description: The values in the metadata are not in the correct format. Action: Ensure that the agent/metadata tool is putting the date and/or number values in the correct format. Valid formats are specified in the Quality Management Administrator User Guide. Failure to update metadata record due to invalid key <key>:<invalid key> Description: The key that the metadata tool is using is not correct according to what is defined in QM Administrator. Action: Check that the metadata tool or the agent (if they enter the key) is using the correct key as defined in QM Administrator. Extension number(s) is/are not in the inclusion list: <extension number(s)>. Extensions changed: <extension number(s)> September 2009 23 Quality Management Troubleshooting Guide Table 6. Audit trail error messages — Continued Message Description/Action Free Space <path, current (Mb)>. Description: Free space checks should occur periodically. The default configuration puts the check at every 18 seconds when free space is less than or equal to 32000 Mb, and every 10 minutes when free space is greater than 32000 Mb. To be approved for upload, free space must be greater than or equal to the file size plus 90% of the failure threshold. The default failure threshold is 1000 Mb. FTS uploads also write to the QM database EventAudit table. FTS has hardcoded warning thresholds of 32000, 16000, 8000, and 4000 Mb. Each time a warning threshold is breached, the appropriate message is written to the EventAudit table and to the fts.log file. Action: None. Description: At the startup of the QM Tomcat service on the Voice and Screen servers, if FTS is unable to run the free space check, it turns off future free space checks. Action: Troubleshoot why FTS was unable to run the free space check. Description: FTS Free space checking has been turned off by the user. By default, it is enabled. Action: FTS will no longer check free space for storage on the Voice and Screen servers. As a result, these servers might fill up until there is no more room for more recordings. The free space check should be enabled. Description: A prior free space warning has been cancelled. Action: None. Description: A QM user with the specified version of QM Desktop Recording service installed has logged in. Action: None Free Space: Checking free space failed. Free space checking has been disabled. Free Space: Checking free space has been manually turned off. Free Space: Prior Warning Cancelled. <path, current (Mb)>. Logged in with version: <current version>. 24 September 2009 Audit Trail Error Messages Table 6. Audit trail error messages — Continued Message Description/Action Logged out. Description: A QM user has logged out. Action: None. Description: The MAC address for the agent’s phone is not associated with the JTAPI user. No recording will take place. Action: Be sure the phone is correctly configured in Cisco Unified Communications Manager (Unified CM). Verify the agent’s phone is associated with the QM JTAPI user. Description: The QM Desktop Recording service is connected to the QM CTI service. Action: None. Description: The QM Desktop Recording service is not connected to the QM CTI service. No recordings will take place. Action: Be sure the QM CTI server IP address is set correctly in QM Administrator and that the QM CTI service is running. Verify the JTAPI user name and password are correct for the configured QM JTAPI user. Description: The QM Desktop Recording server is ready to record. Action: None. Description: The number of recorded voice files ready to be uploaded. Action: None. Recorded (voice/screen): <number of voice and screen files> Description: The number of recorded voice and screen files ready to be uploaded. Action: None. Screen recording failed to start. Description: Screen device failed to start recording. Action: Restart the QM Desktop Recording service; systems will initialize the screen device. Verify the PROXY Pro Host service is running. Restart the PROXY Pro Host service. MAC address is not associated with the JTAPI user: <MAC address>. QM CTI service is connected: <CTI Server IP address> QM CTI service is disconnected: <CTI Server IP address> QM recording software is ready to record. Recorded (voice): <number of voice files>. September 2009 25 Quality Management Troubleshooting Guide Table 6. Audit trail error messages — Continued Message Description/Action Service is stopped while user has not logged out: <user name>. Description: The QM Desktop Recording service is stopped while the user is still logged in. No recording will take place. Action: Restart the QM Desktop Recording service or reboot the computer. Description: The specified user does not belong to a team. No recording will take place. Action: Assign the user to an ACD or QM team and synchronize the data with QM using QM Administrator to initiate the synchronization process. For quality management recording, verify the team is assigned to a workflow. Description: The QM Desktop Recording service was unable to detect the MAC address for the connected phone. No recording will take place. Action: Be sure the agent’s phone and computer are connected in accordance with the section, “QM Desktop Recording Service Requirements” in the Quality Management Installation Guide. If a phone is no longer detected, QM will send a message through Mana. This user must belong to a team to record: <domain\username>. Unable to retrieve phone information. If the agent’s phone and computer are connected correctly and the problem still exists, try the following. Uploaded (voice): <number of voice files> 26 ■ Update the user’s NIC driver. ■ Run a Wireshark capture to verify that the phone is communicating properly with the client PC. Filter the capture for SKINNY (or SIP in the case of a SIP phone) to verify the phone is forwarding its traffic to the client PC. Description: The number of recorded voice files that have been uploaded to the server. Action: None. September 2009 Audit Trail Error Messages Table 6. Audit trail error messages — Continued Message Description/Action Uploaded (voice/screen): <number of voice and screen files> Description: The number of recorded voice and screen files that have been uploaded to the server. Action: None. Uploads Stopped: Not enough free space <path>. Description: FTS has stopped uploading recordings to <path> because the location at <path> is full. Action: Add more storage. Description: The logged-in user is not configured in QM. No recording will take place. Action: Use QM Administrator to configure the user. User is not licensed to record: <user name> Description: The logged-in user is not licensed. Action: Use QM Administrator to license the user and have the user log out and then log back in to begin recording. Voice recording failed to start. Description: Voice device failed to start recording. Action: Restart the QM Desktop Recording service; systems will initialize the voice device. Description: The size of the uploaded file is zero. Action: None. User is not configured to record: <user name>. Zero byte file uploaded: <file name>. September 2009 27 Quality Management Troubleshooting Guide Troubleshooting Issues Installation Issues Problem The client application installations do not download when the links on the installation web page are clicked. Symptom. When the install program link is clicked, a “HTTP 404—File Not Found” error is displayed. Cause. QM Configuration Setup was not completed successfully on the QM server. Solution On the QM server, navigate to the C:\Program Files\Cisco\WFO_ QM\bin folder and double-click PostInstall.exe to launch the QM Configuration Setup utility. If the Tools menu is inactive, QM Configuration Setup is running in Initial mode, indicating that it has not yet been run to completion. If this occurs, navigate to each window and verify that all required data is entered, and then click Finish. If Configuration Setup starts in Initial Mode, it was not completed correctly. Go through each window and make sure that all required data is entered, and then click Finish. If Configuration Setup is running in Update Mode, choose Tools > Generate Info for MSI Clients. This operation regenerates the client installation files in the default location. After the operation completes, try to install the client applications from the installation web pages again. If the problem persists, contact technical support. Problem The message, “Error 1500. Another installation is in progress. You must complete that installation before continuing with this one.” is displayed on a client desktop. Cause. This message can appear when a user attempts to manually upgrade the QM Desktop Recording service after Automatic Updating has already initiated an upgrade. Because the upgrade is running in silent mode the user may not be aware that it is running. 28 September 2009 Troubleshooting Issues Solution Wait five minutes to allow the automatic upgrade to complete. Then check the Add or Remove Programs utility in Control Panel for the version of the installed application. If the version is not correct, manually install the upgrade. Problem While running QM Configuration Setup after installing QM, the following error message is received: “Unable to get information from SQL Server”. You are using SQL Server 2005 and a named instance, not the default instance. Cause. When SQL Server 2005 is installed, it also installs the SQL Browser Service. This service is set by default to be started manually, not automatically. It must be running in order for Configuration Setup to run successfully. Solution Change the SQL Browser Service’s Startup Type from Manual to Automatic in the Windows Services utility in Control Panel, and then run QM Configuration Setup again. If you are upgrading QM from an earlier version. If, after starting the SQL Browser Service, you still receive the error message, call technical support for assistance. If you are installing QM for the first time. If, after starting the SQL Browser Service, you still receive the error message, remove the following files from the C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Data folder: ■ SQMDB.mdf ■ SQMDB_log.LDF ■ SQMREPORTDB.mdf ■ SQMREPORTDB_log.LDF and then run QM Configuration Setup again. IMPORTANT: Do not remove these files if you are upgrading QM. If you do, you will lose data that was present in the previous version of QM. In an initial installation, there is no QM data yet, so these files are empty and can be removed. September 2009 29 Quality Management Troubleshooting Guide QM Configuration Setup Issues Recording Issues Problem Desktop recording fails. Symptom. No recording files are in the C:\Program Files\Common\QM\Recordings folder on the desktop where recording is failing. Solution Action: 30 Check the following possible causes and solutions: ■ Verify that the DNS is configured to resolve IP addresses and host names. From the PC where desktop recording is failing, run the command ping -a <Unified CM IP address>. If the ping cannot resolve the Unified CM IP address, then neither can the QM Desktop Recording service. Configure the DNS so that host names and IP addresses are resolved. ■ Verify that the QM Desktop Recording service is running. ■ Verify that the phone and PC is correctly daisy-chained. The IP phone must be directly connected to the Ethernet jack. The PC must be connected to the phone’s PC port. The phone’s PC port must be enabled. ■ Verify that the client PC is connecting to LDAP by verifying that the sqmservice.dbg has a statement similar to “Connected to LDAP server on <QM LDAP server IP address>.” ■ Verify that the Unified CM with which the phone is registered is entered in the QM Configuration Setup Cisco Unified CM window. ■ Verify that the phone is associated with the QM JTAPI user by verifying that cti.dbg log has the message: “MAC not in domain. <MAC address>.com.spanlink.sqm.ctiservice.CtiException: Specified MAC not in domain: <MAC address>.” and sqmservice.dbg on the agent’s PC will have the message: “The MAC address <phone’s MAC address> is not associated with the JTAPI user.” ■ Verify that the QM CTI service is connected to the Unified CM CTI Manger by looking into cti.dbg for the message: “SQM CTI Service ready.” ■ Verify that the phone is configured in Unified CM according to the Quality Management Installation Guide. Verify the phone is properly configured in Unified CM according to the September 2009 Troubleshooting Issues Quality Management Installation Guide. If the phone is configured correctly and the problem still exists, try the following. Problem ■ Use a packet capture tool (for example, Wireshark) to verify that the phone is forwarding RTP and phone protocol (SCCP or SIP) traffic to the NIC card on the client PC. ■ Verify the Inclusion List node under Site Configuration in QM Administrator is configured correctly. See the Quality Management Administrator User Guide for more information. Recording drops 5-10 seconds of audio near the beginning of a call. Symptom. When CAD and QM are running on the same machine, the recording is missing audio near the beginning of a call and the recording quality might also be garbled. Solution Verify the DNS is configured to resolve host names of the CAD Recording servers. From the PC where desktop recording is failing, open a command window and enter ping -a <CAD Recording server hostname>. If the ping cannot resolve the CAD Recording server hostname, then neither can the QM Desktop Recording service. Configure the DNS so that host names are resolved. Problem Playback of screen recordings fail. Voice recordings are unaffected. The system suffered a power failure but was restarted successfully. Solution The path of the video folder on the Screen server changed to an incorrect location. For example, Configuration Setup displays the screen recording storage path as C:\Program Files\Common Files\QM\recordings\video when they are actually located at E:\Program Files\Common Files\QM\recordings\video. Correct the folder path to the true location in Configuration Setup and screen recordings will play back. If the path is configured correctly and the problem still exists, try the following. September 2009 ■ Stop, then start the PROXY Pro Host service on the QM service. ■ Map a drive to the recording storage location from the client running QM Desktop. If this fails, there might be a policy restriction on the user’s Windows account. 31 Quality Management Troubleshooting Guide Problem The message, “Conversion from raw to spx failed” is seen multiple times in the System Status report, but all the recordings for the specified agent for that day were uploaded correctly. Solution If the staging process that occurs after the configured End Of Day is interrupted before it is completed, you might see this error message in the System Status report. Part of the staging process involves converting the *.raw files to *.spx files. The *.spx files are moved to the Staging folder, while the *.raw files are deleted from the Daily folder. If the staging process is interrupted after some files have been converted and the *.raw files have been deleted, when the staging process resumes, it starts from the beginning so that it appears to fail converting the files that it already processed correctly before the interruption. Examples of interruption include rebooting the PC or restarting the QM Desktop Recording service during the staging process. In these scenarios, no recordings are lost. 32 Problem Unable to record calls from a SIP phone right after rebooting the PC. Solution To detect the connected IP phone, the QM Desktop Recording service monitors the heartbeat messages between the Unified CM and the IP phone. It may take up to 6 minutes after the QM Desktop Recording service starts to properly identify a SIP phone. Problem When playing a contact recording for evaluation, audio and video streams are out of sync. Solution Stop and then restart the playback to resync the audio and video. If that does not work, click the Cancel button and exit the evaluation form, then open it again and start over. September 2009 Troubleshooting Issues Problem In QM Desktop, an agent’s recordings show up in the Archives tab but do not appear on the Recordings tab. Solution The agent is not part of a workflow. Agents must be part of a workflow in order for their recordings to appear on the Recordings tab. Their recordings do show up on the Archives tab as long as archiving is enabled. Problem The QM Network Recording service is stopped while an agent is on a call, and then restarted before the call ends. The call recording only goes until the QM Network Recording service stops, and call duration only includes the time until the QM Network Recording service stops. It does not include time through end-of-call. Solution When the QM Network Recording service is stopped, it drops the connection to the QM CTI service, which in turn drops the connection to JTAPI. The QM CTI service discards all history associated with that client. When a new connection is established, JTAPI provides a snapshot of call events (ringing, established, etc), but does not include RTP events. In the case of the QM Desktop Recording service (the endpoint recording client), it will begin to record again since that service uses only call events. However, the QM Network Recording service (server-based recording) requires RTP events to begin recording. It will start recording on the next RTP event (this could be after a hold on the same call). Problem Playing back a screen recording prompts a username/password popup. This typically happens when using external screen storage. Solution On the screen server, launch PROXY Pro Gateway Administrator (Start > Programs > Proxy Networks > PROXY Pro Gateway Administrator). Navigate to Gateway Security. In the right pane, click the link, Click here to Change Operation Security. On the Operation Security tab, select the Connect to File check box. September 2009 33 Quality Management Troubleshooting Guide Problem When using server-based recording, screen recordings fail. This typically happens when using external screen storage. Solution On the screen server, launch PROXY Pro Gateway Administrator (Start > Programs > Proxy Networks > PROXY Pro Gateway Administrator). Expand the Local Gateway node, and then expand Gateway Server Settings. Select Gateway Security. . In the right pane, click the link, Click here to change Operation Security. On the Operation Security tab, select the Allow check box for the Record to File permission. Problem When using the QM Server API exportRecording operation to perform server-based exporting, no export files are generated. Solution Try the following solutions. ■ Restart the QM Tomcat Service ■ Use QM Desktop to export to wav or wmv files If the problem persists, contact technical support. For more information, see the Quality Management API Programmers Guide. Problem After upgrading from QM 2.4, the screen portion of a recording does not play back, but the audio portion does play back. Solution The upgrade from QM 2.4 to 2.6 or greater does not automatically trigger a required server reboot. To correct the problem, reboot the Screen Recordings server. Problem Contacts are not being uploaded. Symptom. When contacts are not being uploaded, look for the following symptoms. ■ 34 The start time is off by an hour. This can happen when a time zone changes its rules (for example, DST is changed to another date and/or time). September 2009 Troubleshooting Issues ■ The dbproxy.log contains the following message. Unknown Java Timezone ID: null. Endpoint Timezone Cause. Unable to update timezone in the data because the timezone is not in the tzmappings files (for example, Pakistan Standard Time). The Java version shipped with QM does not include the Windows time zone specified in the dbproxy.log. Solution To update the tz database for Java to include your time zone: 1. Download the latest JDK DST Timezone Update Tool (tzupdater) from the following URL. http://java.sun.com/javase/downloads/index.jsp#timezone The tool is bundled in a zip file. 2. Extract the the files from the zip file to a known directory. You can chose to extract the file to the Java\bin folder. If you installed QM in the the default location, the directory path is C:\Program Files\Cisco\WFO_QM\Java\bin. If you do not know where QM is installed, the install location is provided at HKEY_LOCAL_LOCAL_ MACHINE\SOFTWARE\Spanlink\SQM\Site Setup\Install Directory. 3. Select Start > Run, type cmd in the Open field, and then click OK. 4. Enter the following command. cd <directory-path> Where <directory path> is the location of the Java\bin folder. 5. Enter the following command to check the current Java version against the downloaded tzupdater.jar version. java -jar <known-directory-path>tzupdater.jar -V Where <known-directory-path> is the location of the tzupdateder.jar file. If the tzupdater.jar is located in the Java\bin directory, you do not need to specify the known directory path. These commands are case-sensitive. 6. Enter the following command to update the JRE’s timezone version. java -jar <known-directory-path>tzupdater.jar -U September 2009 35 Quality Management Troubleshooting Guide 7. Enter the following command to verify the current Java version is the same as the downloaded tzupdater.jar version. java -jar <known-directory-path>tzupdater.jar -V If the problem persists, contact customer support. Problem Garbled speech appears in the QM call recording when Cisco Unified CM-based monitoring is used. Symptom. The speech is garbled when trying to silently monitor a conversation. Cause. This type of silent monitoring causes an extra stream to be sent from the phone. The extra stream is captured by the QM recording software (desktop or server-based) and stored in the call recording file. This type of silent monitoring can only be used on Cisco-supported hard phones. Solution Don’t use this method of silent monitoring. QM Administrator Issues Problem On some windows in QM Administrator (for instance, the questions area on the Evaluation Form Templates window), buttons appear cut in half. Symptom. Buttons are not displayed correctly. Cause. The Display DPI setting is set to something other than Normal. 36 Solution In the Windows Control Panel, start the Display utility. On the Settings tab, click Advanced. In the resulting Plug and Play Monitor Properties dialog box, select the General tab and make sure the DPI Setting is set to Normal size (96 dpi). Click OK twice to save and apply your settings. Problem In QM Administrator (Personnel > User Administration > Link Selected Users), Active Directory users are not found if the domain is identified by the host name. When a user is selected, the Link Selected Users dialog is launched and domain information is displayed. However, when September 2009 Troubleshooting Issues Find is clicked, an error message displays indicating that no data is available. Cause. In Site Configuration, the Active Directory domains were added using the host name to identify the Active Directory connection. The connection was validated and the domain configuration was saved. Solution Edit the domain configuration to change the host name to an IP address. Once changed, Active Directory data can be found in the Link Selected Users window. If you want to continue using host names, add the host name to the DNS path to ensure that the host name is reachable by all computers. Problem QM 2.5 provided the ability to configure who can evaluate all forms from the User Administration window. The ability to configure who can evaluate forms in the User Administration window in QM 2.5 was lost when upgrading to QM 2.7. The settings revert back to default settings. Solution QM 2.7 or greater now allows you to configure who can evaluate a new form when it is created or modified. The evaluation and approval settings check boxes appear on the Properties tab in the Evaluation Form Administration window. Problem Not enough calls are saved for quality management workflows. Symptom. Calls that should be saved for quality management workflows only appear under the Archive tab in QM Desktop when an agent shuts down or restarts their machine at the end of the day. Solution September 2009 Either set the End of Day time to a later time in QM Administrator or ensure that the agent’s machine is not shut down or restarted at the end of the day. 37 Quality Management Troubleshooting Guide QM Desktop Issues Problem Reports in CSV, PDF, and XLS format do not open in Microsoft Internet Explorer 7. Symptom. A dialog to save the report opens but closes again very quickly. Solution Clear the Confirm open after download check box for the CSV, PDF, and XLS file types. To do this, follow these steps for each file type: 1. Double-click My Computer. 2. On the Tools menu, choose Folder Options. 3. Select the File Types tab. 4. Under Registered File Types, select the file type, and then click Advanced. 5. Clear the Confirm open after download check box, and then click OK. Problem An agent can view calls on the Dashboard tab but not on the Recordings tab. Cause. This happens when any of the following has occurred: ■ The agent’s team is not in a group ■ The agent is deleted from Unified CM Solution To see the calls on the Recordings tab, do one of the following (as appropriate to the individual situation): add the team to a group in QM Administrator, or add the agent in Unified CM. Problem The user is unable to log in. Solution Check the following to verify if the Windows account the user is logging in with is the correct account: ■ 38 Check the sqmservice.dbg file for the credentials the user is using to log in. September 2009 Troubleshooting Issues ■ Verify the Active Directory path is configured correctly in QM Administrator under Enterprise Settings. ■ Verify the QM server is in the user’s domain, or in a trusted domain. ■ Verify the user is synchronized in QM, linked, and licensed. ■ Verify the user account in QM was not deactivated and the agent was not removed from Unified CCX. Problem In a conference call, the DNIS is shown as “Conference.” Solution QM uses JTAPI for call events. The Calling Number shown in QM Desktop is the unmodified calling number. It will stay the same throughout the call, no matter if the call is transferred or conferenced. The Called Number shown in QM Desktop is the last route point it went through. If the call was a direct call (or the route points are not being monitored by the JTAPI user defined in Site Configuration in QM Administrator) it will be the unmodified called number. It will stay the same through transfers and conferences. There might be cases where the unmodified calling/called number is unknown due to devices being registered while the call is already in progress. Refer to the JTAPI Guide for your version of Unified CM. Service Issues Problem The Sync service is not synchronizing databases. Solution Ensure that the IP address for both Side A and Side B are correct. The IP address and “side” are tied together and are not interchangeable. For example, you cannot specify the IP address for Side B in the Side A field. Ensure that the user account under Site Configuration > Cisco Unified CC Database > Authentication is the correct NT or SQL user. Verify that the user is configured correctly according to the Quality Management Installation Guide. September 2009 39 Quality Management Troubleshooting Guide 40 Problem When the QM Tomcat service is restarted or stopped, the process gives a warning error message that the system is unable to stop the service. Solution In fact, the QM Tomcat service really is stopped at that point. It was shut down before it was fully initialized. The user is able to manually restart the QM Tomcat service cleanly. September 2009 Troubleshooting Flowcharts Troubleshooting Flowcharts The following flowcharts explain how to diagnose and correct problems that might occur after QM is installed. The flowcharts cover the following problems. ■ The QM Desktop Recording Service is not Recording Audio (page 41) ■ The QM Desktop Recording Service is not Recording the Screen (page 48) ■ QM Desktop does not Play Back Recordings (page 50) ■ The QM Network Recording service is not Recording Client Calls (page 51) The QM Desktop Recording Service is not Recording Audio Use the following flowchart to resolve the problem when QM Desktop Recording service is not recording audio. September 2009 41 Quality Management Troubleshooting Guide Figure 4. 42 QM Desktop Recording service is not recording audio September 2009 Troubleshooting Flowcharts Figure 4. September 2009 QM Desktop Recording service is not recording audio — Continued 43 Quality Management Troubleshooting Guide Figure 4. 44 QM Desktop Recording service is not recording audio — Continued September 2009 Troubleshooting Flowcharts Figure 4. September 2009 QM Desktop Recording service is not recording audio — Continued 45 Quality Management Troubleshooting Guide Figure 4. 46 QM Desktop Recording service is not recording audio — Continued September 2009 Troubleshooting Flowcharts Figure 4. September 2009 QM Desktop Recording service is not recording audio — Continued 47 Quality Management Troubleshooting Guide The QM Desktop Recording Service is not Recording the Screen Use the following flowchart to resolve the problem when QM Desktop Recording service is not recording the screen. Figure 5. 48 The QM Desktop Recording service is not recording the screen September 2009 Troubleshooting Flowcharts Figure 5. September 2009 The QM Desktop Recording service is not recording the screen — Continued 49 Quality Management Troubleshooting Guide QM Desktop does not Play Back Recordings Use the following flowchart to resolve the problem when QM Desktop does not play back recordings. Figure 6. 50 QM Desktop does not Play Back Recordings September 2009 Troubleshooting Flowcharts The QM Network Recording service is not Recording Client Calls Use the following flowchart to resolve the problem when the QM Network Recording service is not recording client calls. Figure 7. September 2009 The QM Network Recording service is not Recording Client Calls 51 Quality Management Troubleshooting Guide Figure 7. 52 The QM Network Recording service is not Recording Client Calls September 2009 Index A Audit trail error messages 23 C Configuration files 13 ContactDelete utility 18 R Recording storage location moving 19 Registry entries Ident 8 site setup 7 Related documentation 5 S D Debugging 10 disabling 16 editing configuration files 13 enabling 14 Service names and executables 6 T Troubleshooting 28 E Executable files 6 L Logs 10 editing configuration files 13 message formats 12 M Moving the recording storage location 19 Q QM documentation 5 QM services 6 September 2009 53 Quality Management Troubleshooting Guide 54 September 2009