#CLUS
Solution Troubleshooting
for UCCE
Mike Chen
Technical Leader, CJSBU – Solution Assurance
BRKCCT-2210
#CLUS
I assume the following…
•
You have a general or functional understanding of the following:
o
1xxx
2xxx
3xxx
UCCE, CVP, PG, AW, CUCM, ECE, Finesse, etc.
•
You’ve been involved with troubleshooting UCCE in some way.
•
For some of you, I can invoke an instant “OMG, I sooo want to geek
out right now” reaction when I say things like:
o
ICM | GeoTel | Hotfixes | Windows NT
o
Dumplog, opctest, procmon, exit_router, exit_opc, EMSTraceMask
LTR, TEC, BRK
Our Goal: Learn. Collaborate. Have fun!
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
3
Agenda
•
“Trouble, Say Hello To My Little Friend…”
o Root Cause Analysis (RCA)
•
Shooting Trouble in a Complex Solution and Environment
o Monitoring and Response Readiness – Giving RCA a Fighting
Chance
o Root Cause Multipliers: Data Collection Tools and Methods
•
Fault Isolation for Common, Challenging UCCE Operations
o Perspectives from Customers and Agents
o Deconstructing Blended, Omnichannel Routing and Flows
•
Connecting The Dots: Correlation and Causation
•
Summary and Q&A
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
4
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
SPEAKER 1
How
SPEAKER 2
1 Find this session in the Cisco Live Mobile App
WEBEX TEAMS
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
DOCUMENTS
4 Enter messages/questions in the team space
Webex Teams will be moderated
by the speaker until June 16, 2019.
cs.co/ciscolivebot#BRKCCT-2210
#CLUS
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
5
The Contact Center Enterprise – 20,000ft View
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
6
The Contact Center Enterprise Solution
•
Consists of multiple independent sub-components
o
Windows Servers
o Microsoft SQL database
o
Cisco VOS (Voice Operating System)
o IBM Informix database
•
The solution has ~30 core and ~10 optional .exe’s
•
UCCE is a real—time, mission critical application
•
UCCE Solution stability is highly dependent on the network!
•
o
Public RTT >80ms | >200ms | >400ms latency and the solution may get “sassy”
o
Private RTT >500ms latency and the solution will “let you know” by invoking
High-Availability failover between side A and side B components.
Omni-channel customer contact routing across an enterprise blended
agent environment à sensitive and reactive endpoints often becomes your
early warning system.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
7
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
8
The Contact Center Enterprise Solution
•
Consists of multiple independent sub-components
o
Windows Servers
o Microsoft SQL database
o
Cisco VOS (Voice Operating System)
o IBM Informix database
•
The solution has ~30 core and ~10 optional .exe’s
•
UCCE is a real—time, mission critical application
•
UCCE Solution stability is highly dependent on the network!
•
What’s the significance
of those numbers?
o
Public RTT >80ms | >200ms | >400ms latency and the solution may get “sassy”
o
Private RTT >500ms latency and the solution will “let you know” by invoking
High-Availability failover between side A and side B components.
Omni-channel customer contact routing across an enterprise blended
agent environment à sensitive and reactive endpoints often becomes your
early warning system.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
9
Enabling
Root Cause Analysis
The “Big Bang” Theory of Root Cause Analysis
•
Panic, confusion, and noise.
•
Assigning blame, finding fault:
•
o
“Who’s fault is it?” Must be a collaboration and not an effort to blame
o
“Lets first fix the issue; we can all blame Mike later.”
Weak Problem Descriptions (sampled from *actual* Cisco TAC cases)
“agents restarted and no server”
•
Not enough logs collected
•
Not enough debugs enabled
•
Wrong set of logs collected
•
Wrong timeframe collected.
“getting fin error”
“lost calls and some errors”
Logs have been overwritten
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
11
A Defect and ES Patch Dashboard
•
Your one-stop shop for (mostly) current know defects and available patches
https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/cont
act_center/icm_enterprise/EngineeringSpecials/rcct_r_cc-engineeringspecials-htmllandingpage.html
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
12
A Defect and ES Patch Dashboard
•
Your one-stop shop for (mostly) current know defects and available patches
https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/cont
act_center/icm_enterprise/EngineeringSpecials/rcct_r_cc-engineeringspecials-htmllandingpage.html
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
13
On-Line Log Analysis Utility
•
Collaboration Solutions Analyzer (Cisco.com account required)
https://cway.cisco.com/docs/tools/CollaborationSolutionsAnalyzer/csa.calle
ngine.about.html
•
Supported platforms (CUCM, CVP, Jabber, etc.)
https://cway.cisco.com/docs/tools/CollaborationSolutionsAnalyzer/csa.calle
ngine.supportedproducts.html
•
List of tools
https://scripts.cisco.com/app/CallEngineFrontend/#/tool/none
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
14
On-Line Log Analysis Utility
Creates custom, sharable URL
Creates custom, sharable URL
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
15
On-Line Log Analysis Utility
•
Collaboration Solutions Analyzer is not a substitute to manual log analysis
but it does increase root cause analysis velocity.
•
Default logging is sufficient in most cases.
•
Results are sharable with colleagues (must have Cisco.com login)
•
Not just used when things go wrong. A great supplement in pre-deployment
pre-production as well!
•
Developed by Cisco TAC and you can submit feedback
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
16
Understanding UCCE’s Serviceability Framework
•
UCCE’s SNMP, syslog, debugging, logging and storage is managed by the
Event Management System (EMS)
•
Log file location: [drive]:\icm\<instance>\<node>\logfiles
•
File name: Process_YYMMDD_HHMMSS.ems
•
Log files are stored as application binary. Must use UCCE serviceability tools
to capture.
•
When it comes to debugging, all logfile tracing is set to Debug Level 0 by
default; errors and warning trace messages safe for normal operation.
o
•
Router = ra / rb
Logger = la / lb
Peripheral Gateway = PG1A, PG1B
PG2A, PG2B
Admin Workstation = aw / dis / cms
Four log levels 0 – 3. Higher the number, the more risk to performance.
Most log files leverage
for real-time file compression.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
17
Understanding UCCE’s Serviceability Framework
•
Baseline log retention across ALL UCCE nodes
o
Sort by date/time and note max/min timestamps
Processes (.exe)
Retaining ~10 days
(Router & Logger)
•
Central Controller core processes: rtr, lgr, mds, ccag
•
Peripheral Gateway core processes: opc, mds, pim, pgag
•
Adjusting log retention in UCCE is a bit of “adjust and observe”
#CLUS
BRKCCT-2210
Key critical logs
Where heavy
logging occurs
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
18
Understanding UCCE’s Serviceability Framework
•
Log retention management controlled via registry values
•
Registry hive: HKLM\SOFTWARE\Cisco Systems,
•
Registry keys:
Inc.\ICM\<Instance>\<Component>\EMS\CurrentVersion\Library\Processes\<process>\
o
EMSLogFileMax = Maximum bytes allowed for a single log file.
o
EMSLogFileCountMax = Maximum number of log files allowed for a process.
o
EMSAllLogFilesMax = Maximum (combined size) in bytes allowed for all log files of
a process.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
19
Understanding UCCE’s Serviceability Framework
•
Finesse logs do not tend to over-write as aggressively or frequently as
UCCE logs.
•
Increasing debugging on the Finesse servers is rarely required.
o
•
Only enable debugging under TAC guidance.
Access Finesse servers logs from: https://<address>/finesse/logs
Log Retention
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
20
Monitoring and Telemetry Capabilities
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
21
Data Monitoring and Telemetry Capabilities
•
UCCE MIB provides several alert states
•
Single-state notifications
o
•
Example: % of (DB) free space,
peer disconnect
Dual-state notifications
o
Example: Process (.exe) cycling
o
Correlation required
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
22
Data Monitoring and Telemetry Capabilities
•
The UCCE solution leverages Simple Network Management Protocol
(SNMP) for actionable, health impacting notification and syslog for more
granular event logging.
•
UCCE SNMP MIBs (<logger>:\icm\snmp)
o
•
CVP SNMP MIBs (<drive>:\Cisco\CVP\conf)
o
•
CISCO-CONTACT-CENTER-APPS-MIB
Alerting-Based
(trap)
ciscoCcaMIBNotifs(0).cccaIcmEvent(1)
1.3.6.1.4.1.9.9.473.0.1
CISCO-CVP-MIB
Statistics-Based
(polling)
Finesse MIBs
o
CISCO-VOICE-APPS-MIB
o
HOST-RESOURCES-MIB
o
SYSAPPL-MIB
o
MIB-II
VOS Platform
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
23
Data Monitoring and Telemetry Capabilities
•
UCCE’s Simple Network Management Protocol (SNMP) Framework
Event Management System
Syslog
Collector
syslog
PG
Logger A
Router
SNMP
1.3.6.1.4.1.9.9.473.0.1
SNMP
Network
Management
System
OID
AW
t_Event table in Logger DB
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
24
Data Monitoring and Telemetry Capabilities
• Poling SNMP MIB Capabilities – Keeping A Pulse On Your Contact Center
CCE
CVP
Download
12.0 CCE MIB
12.0 CVP MIB
https://tinyurl.com/UCCE-12-0-MIB
https://tinyurl.com/CVP-12-0-MIB
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Data Monitoring and Telemetry Capabilities
•
CVP MIB provides statistical data
•
service ccvp{service}{type}{name}
o
•
-- ******************************************************************
-- Port Licensing Information
-- ******************************************************************
Service identifier: “sip” “ivr” “icm” “reporting” “vxml”
type ccvp{service}{type}{name}
o
“Rt” = Real Time (ccvpSipRtActiveCalls)
”Int” = Interval (ccvpSipIntNewCalls)
o
“Agg” = Aggregate (ccvpSipAggNewCalls)
o
ccvpLicRtPortsAvail OBJECT-TYPE
SYNTAX
Gauge32
UNITS
"ports"
MAX-ACCESS
read-only
STATUS
current
DESCRIPTION
"The real-time port licenses available object is a real-time
snapshot metric indicating the number of port licenses
available for the processing of new calls. Exactly one port
license is used per call, independent of the calls traversal
of the individual call server subsystems."
::= { ccvpLicensingInfo 1 }
ccvpLicRtPortsInUse OBJECT-TYPE
SYNTAX
Gauge32
UNITS
"ports"
MAX-ACCESS
read-only
STATUS
current
DESCRIPTION
"The real-time ports in use object is a real-time snapshot
metric indicating the number of port licenses currently in
use on the call server. Exactly one port license is used
per call, independent of the call's traversal of the
individual call server subsystems."
::= { ccvpLicensingInfo 2 }
ccvpLicRtPortState OBJECT-TYPE
SYNTAX
INTEGER {
safe(0),
unknown(1),
warning(2),
critical(3),
failure(4)
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
26
Data Monitoring and Telemetry Capabilities
•
Core UCCE servers run on Windows OS
•
Bulk of serviceability instrumentation available
via Windows Performance Monitor
•
Full list of Performance Counters in
serviceability guide (chapter 7):
https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/c
ust_contact/contact_center/icm_enterprise/icm_enterprise
_12_0_1/Configuration/Guide/ucce_b_12-serviceabilityguide.pdf
•
Refer to Appendix A for FULL list of UCCE
counters.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
27
Data Monitoring and Telemetry Capabilities
•
Counters you should know at all times…
ü Router
ü Peripheral
(system aggregate)
CccaRouterEntry ::= SEQUENCE {
cccaRouterCallsPerSec
Gauge32,
.1.3.6.1.4.1.9.9.473.1.3.1.1.2 (OID)
cccaRouterAgentsLoggedOn Gauge32,
.1.3.6.1.4.1.9.9.473.1.3.1.1.3 (OID)
cccaRouterCallsInProgress Gauge32,
.1.3.6.1.4.1.9.9.473.1.3.1.1.4 (OID)
cccaRouterCallsInQueue
Gauge32,
.1.3.6.1.4.1.9.9.473.1.3.1.1.8 (OID)
Congestion Level | Rejection Percentage (PerfMon Only)
Gateway (individual)
CccaPgEntry ::= SEQUENCE {
cccaPgCallsInProgress Gauge32,
.1.3.6.1.4.1.9.9.473.1.3.5.1.7 (OID)
cccaPgAgentsLoggedOn Gauge32,
.1.3.6.1.4.1.9.9.473.1.3.5.1.8 (OID)
cccaPgAgentsReady Gauge32,
.1.3.6.1.4.1.9.9.473.1.3.5.1.9 (OID)
cccaPgAgentsTalking Gauge32,
.1.3.6.1.4.1.9.9.473.1.3.5.1.10 (OID)
CTISvr Sessions Open (PerfMon Only)
CTISvr Total Sessions (PerfMon Only)
ü MDS
peer-to-peer (latency)
MDS Private Net Avg RTT (PerfMon Only)
MDS Private Net Max RTT (PerfMon Only)
MDS Private Net Min RTT (PerfMon Only)
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
28
References
•
Refer to UCCE’s Serviceability Guide for more details
https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_conta
ct/contact_center/icm_enterprise/icm_enterprise_12_0_1/Configurat
ion/Guide/ucce_b_12-serviceability-guide.pdf
•
Cisco’s MIB locator and download:
https://mibs.cloudapps.cisco.com/ITDIT/MIBS/servlet/index
•
A free (non-Cisco) MIB browser. Parse a MIB’s OID tree
http://ireasoning.com/downloadmibbrowserfree.php
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
29
Re-Cap
•
The arch nemesis of root cause analysis: “The logs have overwritten.”
•
Contact centers are dynamic; they grow
over time. Know your current environment.
•
Adjust log retention to fit your deployment.
o
Archive/backup logs if needed.
•
Leverage SNMP, syslog, and Performance
Monitor
•
Communicate clearly
o
Provide clear and detailed problem descriptions.
o
Provide clear and detailed problem scope and impact.
Define expectations.
o
•
Be cognizant of efforts that result in diminishing returns.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
30
Common UCCE
Serviceability
Tools
UCCE Troubleshooting Tools - RTMT
•
Real Time Monitoring Tool (RTMT) is a Javabased client-side application that leverages
HTTPS and TCP to monitor system
performance and status.
•
Designed and optimized for VOS applications
but can also be used to enable UCCE
debugging and to collect logs.
•
Download RTMT via CUIC Administration:
https://<cuic>/oamp
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
32
UCCE Troubleshooting Tools - RTMT
•
Because core UCCE servers run on Microsoft Windows, RTMT’s primary
function is to enable tracing and collect logs via “AnalysisManager”
•
You must connect RTMT to CUCM in order to create UCCE nodes that enable
log management.
Connected to either
Finesse or CUIC
Connected
to CUCM
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
33
UCCE Troubleshooting Tools - RTMT
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
34
UCCE Troubleshooting Tools - RTMT
•
Enabling debug tracing on UCCE nodes and processes
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
35
UCCE Troubleshooting Tools - RTMT
•
Collecting logs from UCCE nodes and processes
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
36
UCCE Troubleshooting Tools – Portico
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
37
UCCE Troubleshooting Tools – Portico
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
38
UCCE Troubleshooting Tools – Portico
•
Every UCCE (Windows-based) node contains a shortcut to launch the
Diagnostic Framework Portico which is a web-based UI.
•
Portico is only applicable for UCCE Windows-based servers and leverages
the Diagnostic Framework API service that’s installed on every UCCE
Windows-based server.
•
Can only interrogate one node at a time but one user account may access
multiple UCCE nodes at the same time.
•
•
Be sure to specify the domain when logging in!
Each node has its own address. Example:
o
https://<hostname of UCCE node>:7890/icm-dp/rest/DiagnosticPortal/GetMenu
o
Must specify side A or side B
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
39
UCCE Troubleshooting Tools – Portico
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
40
UCCE Troubleshooting Tools – Portico
Click here
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
41
UCCE Troubleshooting Tools – Portico
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
42
UCCE Troubleshooting Tools – Unified System CLI
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
43
UCCE Troubleshooting Tools – Unified System CLI
•
Every UCCE (Windows-based) node contains a shortcut to launch the Unified
System CLI command-line utility.
•
System CLI is a powerful, multi-platform serviceability tool. It combines the
log capture capabilities of both RTMT and Portico.
•
It can pull logs from the following devices:
YouTube Search “ciscocc system cli” for Demo!
http://www.youtube.com/watch?v=iqHZzM_tank
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
44
UCCE Troubleshooting Tools – Unified System CLI
•
System CLI requires no configuration if you are running default ”local” mode.
o
•
In local mode, CLI can only interrogate the local server on which you’re running it
from.
System CLI can also operate in “system” mode allowing you to interrogate
multiple nodes at once while operating the CLI on a single CCE server.
o
No, you cannot “copy” the CLI to a local desktop and run it from there.
o
There’s a specific requirement to run the Java keystore to operate the CLI.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
45
UCCE Troubleshooting Tools – Unified System CLI
We interrupt this program
for an IMPORTANT
PUBLIC SERVICE
ANNOUNCEMENT
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
46
UCCE Troubleshooting Tools – Unified System CLI
•
For System CLI to operate in system mode when running on a UCCE node, it
requires the configuration of a “devices.csv” file for CLI system access.
•
The devices.csv file is a “hostfile” for the CLI to access other nodes outside
of the local UCCE system node it is being ran on.
•
Notice anything with this text file?
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
47
UCCE Troubleshooting Tools – Unified System CLI
•
My personal recommendation: Run the CLI from a CVP server
•
The System CLI operates exactly the same regardless if you run it from a
UCCE or a CVP server.
•
CLI on CVP does not require a “devices” hostfile. Instead, you configure
solution servers via the OAMP, which operates a secure database.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
48
UCCE Troubleshooting Tools – Unified System CLI
•
Typing “system” in CLI will initiate remote sessions to other nodes
•
A powerful tool when collecting full solution logs: “show tech-support”
o
Collects EVERYTHING for the past 24 hours from all of the configured servers!
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
49
UCCE Troubleshooting Tools – good ‘ol dumplog
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
50
UCCE Troubleshooting Tools - dumplog
•
The dumplog utility converts binary log files written by the UCCE
system processes into a readable (.txt) format.
•
The dumplog utility is invoked from a command prompt directly on
the ICM server but it can also be ran locally on a workstation.
•
How to use dumplog: https://www.cisco.com/c/en/us/support/docs/voice-unifiedcommunications/unified-intelligent-contact-management-enterprise/20406-1.html
#CLUS
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCCE Troubleshooting Tools - dumplog
•
Dumplog – dump multiple processes into a single file
Messages sorted by time
•
Manually copy over other
process .ems files into a single
directory and run dumplog
from there to dump all
processes into a single log.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
52
UCCE Troubleshooting Tools – Summary
•
When it comes to collecting UCCE logs, Cisco TAC recommends…
•
Using RTMT for all VOS components
•
Using Unified System CLI on the CVP server when you need to collect logs from
multiple processes on multiple servers.
•
Using Diagnostic Portico and/or Dumplog for UCCE “surgical strikes”
“Only 2 things are infinite, UCCE storage and the universe… I’m not sure about the former.”
~Albert Einstein (I think)
Please (please) remember to delete your log file captures from the
UCCE server once you are done.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
53
When Things Go
Wrong
The
with
users
~Somewhere on the internet
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
55
Troubleshooting The Contact Center
• Cisco’s mantra: “Build bridges, not islands.” | “Be a bridge to possible.”
• Our troubleshooting mantra: “Be a bridge to collaboration.”
• You can’t solve every thing, every time; be an effective bridge to Cisco TAC
• Establish a SOLID baseline.
o Clear, comprehensive detailed problem descriptions
with impact statements.
o Accurate software version and patch information.
o Date/time of incident(s).
o Working vs. non-working example (if possible)
• Collect logs – more is always better!
o Provide a cushion of +2 hours before and -2 hours
after incident window.
o Note zero-byte files to avoid confusion and cycles.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
56
Knowing how to deconstruct / reconstruct
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
57
Understanding UCCE’s Serviceability Framework
15:34:13:810 PG1B-opc Trace: CExceptionHandlerEx::GenerateMiniDump -- A Mini Dump File is available at C:\icm\bin\opc.exe_20180908120245973.mdmp.
15:34:13:810 PG1B-opc Trace: CExceptionHandlerEx::Stack -- DBGHELP.DLL or its exported procs not found
15:34:13:811 PG1B-opc Unhandled Exception: Exception code: E06D7363 Microsoft Compiler Generated (Q185294)
Fault address: 756AC41F 01:0000B41F C:\Windows\syswow64\KERNELBASE.dll
Registers:
EAX:0073D1C0
Bug searchable
EBX:F071C37A
ECX:00000003
EDX:00000000
ESI:000014E0
EDI:0073D2B8
CS:EIP:0023:756AC41F
SS:ESP:002B:0073D1C0 EBP:0073D210
DS:002B ES:002B FS:0053 GS:002B
Flags:00000216
•
•
•
•
•
A
course on MiniDumps
ALL UCCE process are capable of generating minidump files.
Minidumps are application crash dumps.
Contains useful signature information to perform basic debugging.
Small and portable.
To read a minidump file you must have both the application
binaries (code | .exe) and symbol files (.pdb) which contains
addresses for debugging.
X86 Call stack:
Address Frame Logical addr Module
756AC41F 0073D210 0001:0000B41F C:\Windows\syswow64\KERNELBASE.dll
0A90ECB8 0073D248 0001:0000DCB8 C:\icm\bin\SHSMP.dll
0A90DF82 0073D298 0001:0000CF82 C:\icm\bin\SHSMP.dll
001FCD62 0073E75C 0001:0007BD62 C:\icm\bin\opc-cce.exe
0020AB33 0073FC34 0001:00089B33 C:\icm\bin\opc-cce.exe
0020B66A 0073FC50 0001:0008A66A C:\icm\bin\opc-cce.exe
00311DAE 0073FCBC 0001:00190DAE C:\icm\bin\opc-cce.exe
00314FD7 0073FCF8 0001:00193FD7 C:\icm\bin\opc-cce.exe
0021AFBD 0073FD2C 0001:00099FBD C:\icm\bin\opc-cce.exe
7592338A 0073FE04 0001:0 … … …
Application process’ last will and testament
(crash signature)
Process restart
15:34:15:082 PG1B-opc Initializing Event Management System (EMS) Library.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
58
Understanding UCCE’s Serviceability Framework
•
There are two types of minidumps: Small and Full
•
Full dumps contain critical variable and memory index information on the
running routine/method that crashed.
o
By default, UCCE processes are set to small dumps only.
o
You can enable FULL dump for each selective process via the registry.
•
Registry hive: HKLM\SOFTWARE\Cisco Systems,
•
Registry key: EMSGenerateSmallMemoryDump
Inc.\ICM\<instance>\<node>\EMS\CurrentVersion\Library\Processes
o
•
Dynamic key
Recommend enabling full dump for the following UCCE processes: MDS,
OPC, Router, and PIMs
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
59
Understanding UCCE’s Serviceability Framework
Make sure it’s set to ‘1’
To enable FULL dump, update registry value
To end in ‘2’
Example: 0x20000002
Collect the following files:
•
•
•
OPC Example
The .exe and .pdb are
located in “icm\bin”
directory.
Registry path
#CLUS
The .exe that crashed
The .pdb (symbol) file
The .mdmp file
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
60
Understanding UCCE’s Serviceability Framework
•
Open Peripheral Connect (OPC) capture – going beyond logfiles
o
•
•
Binary playback of all OPC events – like MDS, all Peripheral Gateway events go
through OPC
Enabled by default on all Peripheral Gateways at installation
o
Each capture file has a size limit of 10MB
o
Each capture folder has a size limit of 2GB
A powerful root cause aid and supplement that can be used…
o
when Peripheral Gateway logfiles are not available
o
when enabling debugging is too “risky” in a production environment
o
to replicate environmental conditions and aid reproduction
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
61
Understanding UCCE’s Serviceability Framework
•
OPC Capture file path: <drive>:\icm\<instance>\<PGnode>\OPCcap
Max. Folder Size = 2GB (each)
Max. Folder Size = 5GB
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
62
Understanding UCCE’s Serviceability Framework
OPC Capture file size
OPC Capture folder size
Registry path
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
63
Understanding UCCE’s Serviceability Framework
•
Know when OPC Capture files overwrite
•
OPCconfig must match the oldest
OPCdata date/time. Otherwise,
capture files have overwritten.
•
If OPC Capture files have overwritten
o
Initiate ‘newcap’ command via OPCTEST
• How to use OPCTEST
https://www.cisco.com/c/en/us/support/docs/
voice-unified-communications/icm-peripheralgateway/20418-30.html
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
64
UCCE Redundancy and High Availability
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
65
UCCE High Availability
MDS
Enabled Clock
• Public Network (Router ó PG)
Private Network
MDS
Disabled Clock
o Heartbeats sent every 400ms, bidirectional
o 2000ms (2 seconds) latency tolerance:
5 consecutive missed heartbeats x400ms
Router
A
Logger A
Router
B
MDS
Logger B
o Router CCAG.exe ó PG PGAG.exe traffic
• Device Management Protocol (DMP)
Public Network
Active Socket
Public Network
Idle Socket
Number of PG Pairs.
Router’s DMP views PGs as a redundant pair.
MDS
MDS
• Private Network (Peer-to-Peer)
PG1
A
PG1
B
PG2
A
PG2
B
o Heartbeats sent every 100ms, bidirectional
o 500ms latency tolerance:
5 consecutive missed heartbeats x100ms
Private Network
Duplex Mode
o Only Message Delivery System (MDS) peer
synchronizer traffic
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
66
UCCE High Availability
MDS
Enabled Clock
• Central Controller (Router and Logger)
does not have a concept of an “idle”
state. It is always Active / Active.
Router
A
Logger A
o Synchronized processes: MDS, RTR,
Logger
Private Network
MDS
Disabled Clock
Router
B
MDS
Logger B
Public Network
• Peripheral Gateways do maintain Active
/ Idle states.
Active Socket
Public Network
Idle Socket
o Active / Idle process state is apathetic
(doesn’t care) with regard to A or B sides.
o Will not “switch back” post-failover.
o OPC (process) controls where processes
are either active or idle.
PG1
B
PG2
A
PG2
B
Private Network
o MDS and OPC are synchronized, always
Active / Active.
#CLUS
MDS
MDS
PG1
A
Duplex Mode
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
67
Test Other Side (TOS) - Router
MDS
Enabled Clock
• Private Network (Peer-to-Peer)
o Missed 5 consecutive heartbeats
o Latency exceeded 500ms
Private Network
Router
B
Router
A
Logger A
MDS
Disabled Clock
Logger B
• Message Delivery Service (MDS)
12:00:32:273 ra-mds Connectivity with duplexed partner has been lost due to a
failure of the private network, or duplexed partner is out of service.
12:00:32:273 ra-mds Synchronizer switching to non-duplex operation.
12:00:32:273 rb-mds Connectivity with duplexed partner has been lost due to a
failure of the private network, or duplexed partner is out of service.
TOS Requests
Public Network
• Non-enabled MDS will issue a Test
Other Side (TOS) message.
PG1
A
12:00:32:273 rb-mds Initiating test of peer Synchronizer.
12:00:32:273 rb-mds Trace: Sending TOS request: sequence = 1.
PG1
B
PG2
A
PG2
B
Private Network
Isolated / Simplex
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
68
Test Other Side (TOS) - Router
MDS
Enabled Clock
12:00:32:291 rb-ccag Trace: Sending TOS request to device PG01 (seq=1).
Private Network
MDS
Disabled Clock
12:00:32:291 PG1B-pgag Trace: Received TOS request from side B: seq=1.
12:00:32:291 PG1B-pgag Trace: Sending TOS request to side A: seq=1.
Logger A
12:00:32:299 ra-mds Trace: Received TOS request: sequence = 1.
12:00:32:299 ra-mds Trace: Sending TOS response: sequence=1
status=ENABLED.
Router
B
Router
A
12:00:32:299 ra-ccag Trace: Received TOS request from device PG01 (seq 1).
Logger B
12:00:32:299 ra-ccag Trace: Sending TOS response to device PG01: seq=1
status=ENABLED.
12:00:32:302 PG1B-pgag Trace: Received TOS response from side A: seq=1
status=ENABLED.
12:00:32:302 PG1B-pgag Trace: Sending TOS response to side B: seq=1
status=ENABLED.
TOS ACK Path
Public Network
12:00:32:302 rb-ccag Trace: Received TOS response from device PG01: seq=1,
status=ENABLED.
12:00:32:305 rb-mds Trace: Received TOS response: sequence=1
status=ENABLED.
12:00:32:305 rb-mds Peer Synchronizer was found to be active.
12:00:32:305 rb-mds Synchronizer suspending operation.
12:00:32:305 rb-mds MDS going out of service.
12:00:32:305 rb-mds Client: dba, stopping due to error.
12:00:32:305 rb-mds Client: rtr, stopping due to error.
12:00:32:305 rb-mds Client: hlgr, stopping due to error.
12:00:32:305 rb-mds Client: clgr, stopping due to error.
12:00:32:305 rb-mds Client: rts, stopping due to error.
12:00:32:305 rb-mds Client: rtr, registered with handle: 1.
12:00:32:305 rb-mds Client: rtr, started.
PG1
A
PG1
B
PG2
A
PG2
B
Private Network
Isolated / Simplex
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
69
Test Other Side (TOS) – Peripheral Gateway
Private Network
12:22:18:732 PG01A-mds Connectivity with duplexed partner has been lost due
to a failure of the private network, or duplexed partner is out of service.
12:22:18:732 PG01A-mds Synchronizer switching to non-duplex operation.
12:22:18:732 PG01B-mds Connectivity with duplexed partner has been lost due
to a failure of the private network, or duplexed partner is out of service.
12:22:18:732 PG01B-mds Initiating test of peer Synchronizer.
12:22:18:732 PG01B-mds Trace: Sending TOS request: sequence = 1,
pg_weight=120.
Router
B
Router
A
Logger A
Logger B
12:22:18:732 PG01B-pgag Trace: Sending PG TOS request: seq=1,
pgB_weight=120.
12:22:18:782 rb-ccag Trace: Received TOS request from device PG01 (seq 1).
12:22:18:782 PG01A-pgag Trace: Received PG TOS request: seq=1,
pg_weight=120.
PG1
A
12:22:18:792 PG01A-mds Trace: Processing TOS request: sideA_weight=130,
sideB_weight=0, TOS pg_weight=120
12:22:18:792 PG01A-mds Trace: Sending TOS response: sequence=1
status=ENABLED.
PG2
A
PG1
B
MDS
Enabled Clock
PG2
B
MDS
Disabled Clock
Private Network
12:22:18:792 PG01A-pgag Trace: Sending PG TOS response: seq=1
status=ENABLED.
Isolated / Simplex
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
70
Test Other Side (TOS) – Peripheral Gateway
Private Network
12:22:18:792 rb-ccag Trace: Received TOS request from device PG01 (seq 1).
12:22:18:792 rb-ccag Trace: Sending TOS response to device PG01: seq=1
status=ENABLED.
Logger A
12:22:18:792 PG01B-pgag Trace: Received PG TOS response: seq=1
status=ENABLED.
Router
B
Router
A
Logger B
12:22:18:810 PG01B-mds Trace: Received TOS response: sequence=1
status=ENABLED.
12:22:18:810 PG01B-mds Peer Synchronizer was found to be active.
12:22:18:810 PG01B-mds Synchronizer suspending operation.
12:22:18:810 PG01B-mds MDS going out of service.
12:22:18:810 PG01B-mds Client: pim1, stopping due to error.
12:22:18:810 PG01B-mds Client: ctisvr, stopping due to error.
12:22:18:810 PG01B-mds Client: pim3, stopping due to error.
12:22:18:810 PG01B-mds Client: pim2, stopping due to error.
12:22:18:810 PG01B-mds Client: opc, stopping due to error.
12:22:18:810 PG01B-mds Client: ctisvr, registered with handle: 17.
PG1
A
PG2
A
PG1
B
MDS
Enabled Clock
PG2
B
MDS
Disabled Clock
Private Network
Isolated / Simplex
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
71
Peer Recovery
• MDS recovery is initiated when a heartbeat ping is received across the private
network. Even when isolated, MDS continues to send heartbeats to its peer.
• MDS recovery process is the same between Routers and Peripheral Gateways.
• A “flapping” of the private network will trigger multiple occurrences of failover
and recovery.
ra-mds Communication with peer Synchronizer established.
ra-mds Initiating state transfer SEND operation.
ra-mds Trace: Compressed 1415612 bytes into 345416.
ra-mds State transfer operation completed successfully.
ra-mds Synchronizer switching to active duplex operation.
ra-mds Communication with peer Synchronizer established.
rb-mds Initiating state transfer RECEIVE operation.
rb-mds Trace: Objects transferred = 13227, total bytes 1415612.
rb-mds State transfer operation completed successfully.
rb-mds Synchronizer switching to passive duplex operation.
MDS
Router
B
Router
A
Private Network
PG01A-mds Communication with peer Synchronizer established.
PG01A-mds Initiating state transfer SEND operation.
PG01A-mds Trace: Compressed 850755 bytes into 157060.
PG01A-mds State transfer operation completed successfully.
PG01A-mds Synchronizer switching to active duplex operation.
PG01A-mds Communication with peer Synchronizer established.
PG01B-mds Initiating state transfer RECEIVE operation.
PG01B-mds Trace: Objects transferred = 11126, total bytes 850755.
PG01B-mds State transfer operation completed successfully.
PG01B-mds Synchronizer switching to passive duplex operation.
#CLUS
BRKCCT-2210
PG1
A
MDS
PG1
B
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
72
UCCE HA Summary
• Avoid at all cost any possibility of dual (public and private) network outages!
o Dual network instability voids HA benefit
o Can easily trigger UCCE “split-brain”
Achilles’ heel
• For public network disconnects, collect the following process logs from both
sides:
o CCAG from Router | PGAG, OPC, CTIsvr, and PIM from PGs | Finesse server logs
• For private network disconnects, collect the following process logs from both
sides:
o MDS, CCAG, RTR from Router | MDS, PGAG, OPC, CTIsvr, and PIM from PGs
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
73
UCCE Comprehensive Call Flow
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
74
UCCE Call Flow – Component Correlations
CUBE/TDM-IP
Gateway
SIP Invite
Gateway logs
SIP Proxy
1
SIP
CUSP
SIP Invite
1
INVITE sip:2005@192.168.33.11:5060 SIP/2.0 <---------------- Destination CVP
Via: SIP/2.0/UDP 192.168.33.1:5120;branch=z9hG4bKE18A7
Remote-Party-ID: <sip:5007@192.168.33.1>;party=calling;screen=yes;privacy=off
From: <sip:5007@192.168.33.1>;tag=86A2391D-1FB8
To: <sip:2005@192.168.33.11>
Call-ID: EAAE2740-D6B011E7-9E44ED84-EEA4C112@192.168.33.1 <---------------- Switch-Leg Call-ID
Supported: 100rel,timer,resource-priority,replaces,sdp-anat
Min-SE: 1800
Cisco-Guid: 0645482752-0000065536-0000000018-1446391818 <---------------- Cisco-GUID
User-Agent: Cisco-SIPGateway/IOS-15.6.1.S
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO,
REGISTER
CSeq: 101 INVITE
Timestamp: 1512230499
Contact: <sip:5007@192.168.33.1:5120>
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
75
UCCE Call Flow – Component Correlations
Cisco-GUID from Ingress GW
CUBE/TDM-IP
Gateway
SIP Invite
SIP Proxy
1
SIP
CUSP
SIP Invite
2
SIP Service
SIP
ICM Service
CVP logs (C:\Cisco\CVP\logs)
2
1798: 192.168.33.11: Dec 02 2017 17:01:30.952 +0100: %CVP_11_5_SIP-7-CALL: {Thrd=DATAI.10} Found an
incoming CISCO-GUID hdr : 0645482752-0000065536-0000000018-1446391818,
CVP Call-GUID
Generated a new CVP-ID hdr : 2679490000010000000000125636300A
1799: 192.168.33.11: Dec 02 2017 17:01:30.952 +0100: %CVP_11_5_SIP-7-CALL: {Thrd=DATAI.10}
CALLGUID = 2679490000010000000000125636300A LEGID = EAAE2740-D6B011E7-9E44ED84-EEA4C112 [INBOUND]: Display Name [] Is Using Survivability [false] CallServer build CVP_11_6_1_0_1_0_489
1800: 192.168.33.11: Dec 02 2017 17:01:30.952 +0100: %CVP_11_5_SIP-7-PARAM: {Thrd=DATAI.10}
CALLGUID = 2679490000010000000000125636300A LEGID = EAAE2740-D6B011E7-9E44ED84-EEA4C112 [INBOUND]: ReqURI (DN) sip:2005@192.168.33.11:5060 FromURI sip:5007@192.168.33.1
CVP Call Server
CVP Call-GUID
Switch-leg CallID
• The CVP-GUID is the same as the Cisco-GUID. It’s
just written in HEX:
o Dec: 0645482752-0000065536-0000000018-1446391818
o Hex:
26794900
#CLUS
00010000
BRKCCT-2210
00000012
5636300A
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
76
UCCE Call Flow – Component Correlations
CVP logs (C:\Cisco\CVP\logs)
3
SIP Service
ICM Service
425: 192.168.33.11: Dec 02 2017 17:01:30.952 +0100: %CVP_11_5_ICM-7-CALL:
{Thrd=Event-router-group-0} CALLGUID=2679490000010000000000125636300A DialogId=31
[SIP_SWITCH_LEG_DIALOGUE] - Processing from SIP_SS - [NEW_CALL] DialogID=0 SendSeqNo=0 DNIS=2005
ANI=5007 CED= rckey=0 rcday=0 rcseq=0 uui= callguid=2679490000010000000000125636300A trunkGroupId=0
trunkNumber=0 serviceId=0 calledNumber= location= locationpkid= pstntrunkgroupid= sipheader= ECC variables=
ECC arrays= Peripheral Variables=
New Call
UCCE
VRU PG
CVP Call Server
Router
VRU PIMs
New Call
GED-125 (SCI)
3
DMP
426: 192.168.33.11: Dec 02 2017 17:01:30.952 +0100: %CVP_11_5_ICM-7-CALL:
{Thrd=Event-router-group-0} CALLGUID=2679490000010000000000125636300A DialogId=31
[SIP_SWITCH_LEG_DIALOGUE] - Publishing to UCCE - [NEW_CALL] DialogID=31 SendSeqNo=1 DNIS=2005
ANI=5007 CED= rckey=0 rcday=0 rcseq=0 uui= callguid=2679490000010000000000125636300A trunkGroupId=100
trunkNumber=0 serviceId=1 calledNumber= location= locationpkid= pstntrunkgroupid= sipheader= ECC
variables=user.media.id: 2679490000010000000000125636300A, ECC arrays= Peripheral Variables=
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
77
UCCE Call Flow – Component Correlations
4
SIP Service
VRU PIM logs
Message direction
CVP
DialogID
17:02:09:767 pg2a-pim2 Trace: VRU->PG:NEW_CALL(138 bytes):DID=31 SendSeq#=1 TrkGrpID=100 TrkNum=0
SrvID=1 DN=2005
ANI=5007 DNIS=2005 RCK=0 RCKDay=0 RCKSeq#=0 CallGUID=2679490000010000000000125636300A
PstnTrkGrpChann#=0
ICM Service
New Call
4
CVP Call Server
UCCE
VRU PG
Router
VRU PIMs
New Call
OPC
GED-125 (SCI)
DMP
OPC logs
4
Message Sent to Router
CVP
DialogID
17:02:09:768 pg2a-opc Trace:SendINRCMessage:NEW_CALL RCID=5002 PID=5001 DID=31 DIDRelSeq#=1
GrpDelKey=5002 CRS(RtrDate=0,RtrCID=0) RCKSeq#=0
NICCallID={PCID=5001,RCID=5002,Remote=0,0,DID=0x1f,RemDID=0x0,Grp=0,Data=0,RtrData=0,CCID=x00000001/x
00000000} PGCallID={N/A} OperationCode=CLASSIC OperationFlags=COOP_NONE CalledParty#=2005
CallingParty#=5007 CED= MsgOrigin=1 RouteDevType=6 Originator=0 NICCalledParty#=2005 NetDelay=0 Priority=0
OrigType=3 atVRU=T CLIDRestrict=F PrefAGSTID=0 MRDomainID=1 LegacyPreCallMsgEnabled=F CID=0,
ServiceReq=0 CallGUID=2679490000010000000000125636300A PstnTrkGrp(ID=,Chann#=0)
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
78
UCCE Call Flow – Component Correlations
CVP
DialogID
Router log
5
SIP Service
Router CallKeyDay, Router CallKey
Routing Client ID
17:02:09:759 ra-rtr Trace: (31 x 0 : 0 0) NewCall: CID=(152276,314), DN=2005, ANI=5007, CED=, RCID=5002
MRDID=1…
ICM Service
New Call
4
CVP Call Server
UCCE
VRU PG
6
VRU PIMs
New Call
Router
5
GED-125 (SCI)
6
DMP
VRU Label
CorrelationID
Important message!
17:02:09:759 ra-rtr Trace: (31 x 23 : 0 0) TransferToVRU: Label=7777777777, CorID=23, VRUID=5006, RCID=5002
• At this point, the UCCE Router has directed the
call to the VRU with the CVP label.
• The Send to VRU node will generate a
“Temporary Connect” message to CVP.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
79
UCCE Call Flow – Component Correlations
• The “Temporary Connect” message sent to CVP will establish the switch
6
leg of the call. This “holds” the session with CVP for further instructions from
UCCE Router (script).
CVP logs (C:\Cisco\CVP\logs)
427: 192.168.33.11: Dec 02 2017 17:01:30.952 +0100: %CVP_11_5_ICM-7-CALL: {Thrd=Event-router-group-0}
CALLGUID=2679490000010000000000125636300A DialogID= 31 [SIP_SWITCH_LEG_DIALOGUE] –
Processing from UCCE - ,, [ICM_TEMPORARY_CONNECT], rckey=314, rcday=152276, DNIS=2005, ANI=5007
SIP Service
ICM Service
Switch Leg
6
CVP Call Server
Temp
Connect
UCCE
VRU PG
6
6
VRU PIMs
Router
Temp-connect
GED-125 (SCI)
DMP
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
80
UCCE Call Flow – Component Correlations
• Immediately after the “Temporary Connect” message, UCCE (Router) will
send a “Connect” (TransferConnect sent. Dialog pending.) message. This
establishes the VRU leg and a SIP Invite is sent to the VXML GW.
7
CVP logs (C:\Cisco\CVP\logs)
428: 192.168.33.11: Dec 02 2017 17:01:30.952 +0100: %CVP_11_5_ICM-7-CALL: {Thrd=Event-router-group-0}
CALLGUID=2679490000010000000000125636300A DialogID= 31 [SIP_SWITCH_LEG_DIALOGUE] –
Publishing to SIP_SS - [CONNECT] DialogID=31 SendSeqNo=0 Label=7777777777 TxrHint=false rckey=314 rcday=152276 rcseq=0 location= locationpkid=
sipheader= uui= whisperAnnounce=null callguid=2679490000010000000000125636300A ECC variables=user.media.id:
2679490000010000000000125636300A, ECC arrays=USER_MICROAPP_TOEXTVXML[0]:application=customhw, Peripheral Variables=
1806: 192.168.33.11: Dec 02 2017 17:01:30.952 +0100: %CVP_11_5_SIP-7-CALL: {Thrd=pool-1-thread-141-SIP-1481}
VXML GW or VVB
Static route matched 777777777723 to StaticRoute: patterns like 777* will route to 192.168.33.27
VRU Leg
1812: 192.168.33.11: Dec 02 2017 17:01:30.968 +0100: %CVP_11_5_SIP-7-CALL: {Thrd=pool-1-thread-141-SIP-1481}
CALLGUID = 2679490000010000000000125636300A LEGID = 2679490000010000000000125636300A-151223049096833 - [OUTBOUND]: INVITE TO
<sip:777777777723@192.168.33.27;transport=tcp> FROM 5007 <sip:5007@192.168.33.11:5060> EXPIRES[60] 100REL[Unsupported]
SIP Proxy
CUSP
VXML
Gateway
Asks for VXML GW
address
SIP Service
ICM Service
SIP
VRUInvite
Leg
CVP Call Server
7
Connect
UCCE
VRU PG
7
VRU PIMs
Router
Connect
GED-125 (SCI)
DMP
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
81
UCCE Call Flow – Component Correlations
• If you are running Cisco’s Virtual Voice Browswer (VVB), here’s what the logs
will look like.
• VVB has A LOT of different process logs but the only one you have to worry
about is the MIVR log.
Cisco VVB logs (SIP connect)
23717: Dec 02 17:02:30.061 CET %MIVR-JASMIN-7-UNK:[CALLID=2679490000010000000000125636300A-151223049096833] VRU-Leg
NEW_CALL, GUCID=2679490000010000000000125636300A; GUID=0645482752-0000065536-0000000018-1446391818;
CallId=2679490000010000000000125636300A-151223049096833@192.168.33.11; URI=sip:777777777723@192.168.33.27;transport=tcp
………
SIP msg
……….
23881: Dec 02 17:02:30.130 CET %MIVR-SS_SIP-7-UNK:Call.answered() SIPCallContact[id=65,type=Cisco SIP
Call,implId=2679490000010000000000125636300A-151223049096833@192.168.33.11,
active=true,state=CALL_ANSWERED,inbound=true,handled=false,locale=en_US,aborting=false,app=App[name=Comprehensive,type=Cisco Script
Application,id=1,desc=Comprehensive Application,enabled=true,max=66,valid=true,cfg=[ApplicationConfig[schema=ApplicationConfig,time=2017-11-08
11:49:17.0,recordId=2,desc=Comprehensive Application,name=Comprehensive,type=Cisco Script
Application,id=1,enabled=true,sessions=66,script=SSCRIPT[CVPComprehensive.aef],defaultScript=,vars=null,defaultVars=null]]]
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
82
UCCE Call Flow – Component Correlations
Cisco VVB logs (HTTP)
[CALLID=2679490000010000000000125636300A-151223049096833] Invoke:
[CALLID=2679490000010000000000125636300A-151223049096833] Log:
[CALLID=2679490000010000000000125636300A-151223049096833] Log:
[CALLID=2679490000010000000000125636300A-151223049096833] Log:
[CALLID=2679490000010000000000125636300A-151223049096833] Log:
[CALLID=2679490000010000000000125636300A-151223049096833] Log:
[CALLID=2679490000010000000000125636300A-151223049096833] Log:
[CALLID=2679490000010000000000125636300A-151223049096833] Log:
[CALLID=2679490000010000000000125636300A-151223049096833] Log:
[CALLID=2679490000010000000000125636300A-151223049096833] Log:
[CALLID=2679490000010000000000125636300A-151223049096833] Fetch:
Documents/System/bootstrap.vxml
**HTTPS**false**Starting Application...
**CVP Header -- Primary: **192.168.33.11**
**CVP Header -- Port: **7000**
**CVP Header -- Secured Port: **7443**
**CVP Header -- DNIS **777777777723**
**CVP Header -- BACKUP ****
**CVP Header -- CallType: **undefined**
**CVP Header -- PrimaryVXMLServer: **undefined**
**CVP Header -- BackupVXMLServer: **undefined**
http://192.168.33.11:7000/CVP/Server?MSG_TYPE=CALL_NEW&
CALL_DNIS=777777777723&
CALL_ANI=sip%3A5007%40192.168.33.11%3A5060&
ERROR_CODE=0&
RECOVERY_VXML=crtp%3A%2FSDocuments%2FRecovery.vxml&
CLIENT_TYPE=IOS&
CALL_ID=2679490000010000000000125636300A&
CALL_LEGID=2679490000010000000000125636300A-151223049096833%40192.168.33.11&
CALL_UUI=&
VERSION=Voice_Browser_11_6&
PRIMARY=192.168.33.11
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
83
UCCE Call Flow – Component Correlations
• A “Request Instruction” is sent from CVP to UCCE’s Router (via the VRU PG).
The “Request Instruction” is CVP’s way of saying: “OK, I’ve got the call but
what do you want me to do with it?”
CVP logs (C:\Cisco\CVP\logs)
8
…CALLGUID=2679490000010000000000125636300A
DialogId=32 [IVR_VRU_LEG_DIALOGUE]
- Processing from IVR_SS - [NEW_CALL] …
…CALLGUID=2679490000010000000000125636300A DialogId=32 [IVR_VRU_LEG_DIALOGUE] - Publishing to UCCE - [REQUEST_INSTRUCTION] DialogID=32
SendSeqNo=1 ANI=5007 DNIS=777777777723 correlationId=23 trunkGroupId=200 trunkNumber=0 serviceId=2 uui= calledNumber= location= locationpkid=
pstntrunkgroupid= pstntrunkgroupchannelnum=0 sipheader=
Correlation ID
8
Router logs
17:02:09:869 ra-rtr Trace: (31 32 23 : 0 0) RequestInstr: CID=(152276,314), CallState=1
Switch Leg VRU Leg
SIP Service
ICM Service
Router
CallKey Day
Router
CallKey
SIP
VRUInvite
Leg
8
Request
Instruction
UCCE
VRU PG
8
VRU PIMs
CVP Call Server
Router
Request
Instruction
GED-125 (SCI)
DMP
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
84
UCCE Call Flow – Component Correlations
• At this point, you are executing the Script logic within UCCE.
• Depending on your call treatment, you will see multiple
“Request Instructions” from VXML GW to CVP to the UCCE
Router. These will come after every successful “Instruction”
execution.
• By default, we do not trace UCCE Router steps in the Router
logs. You can enable this by typing “rtrtrace” in cmd prompt.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
85
UCCE Call Flow – Component Correlations
23:45:12:145 ra-rtr Trace: Script-Execute CID=(152276,314) Default\\UCLAB_INB_SCRIPT_027302 If 43
23:45:12:145 ra-rtr Trace: Script-Execute CID=(152276,314) Default\\UCLAB_INB_SCRIPT_027302 Set Variable 48
23:45:12:145 ra-rtr Trace: Script-Execute CID=(152276,314) Default\\UCLAB_INB_SCRIPT_027302 Run External Script 49
23:45:12:145 ra-rtr Trace: Script-Execute CID=(152276,314) Default\\UCLAB_INB_SCRIPT_027302 Run External Script 50
23:45:12:145 ra-rtr Trace: Script-Execute CID=(152276,314) Default\\UCLAB_INB_SCRIPT_027302 Run External Script 51
23:45:12:145 ra-rtr Trace: Script-Execute CID=(152276,314) Default\\UCLAB_INB_SCRIPT_027302 Set Variable 52
23:45:14:230 ra-rtr Trace: Script-Execute CID=(152276,314) Default\\UCLAB_INB_SCRIPT_027302 Queue Target 73
23:45:59:472 ra-rtr Trace: Agent 13851 (149795) on peripheral 5001 (CUCM_PG_2) logged in to group 5001.
23:45:59:472 ra-rtr Trace: Agent 13851 (149795) on peripheral 5001 (CUCM_PG_2) logged in to group 5003.
23:45:59:472 ra-rtr Trace: Agent 13851 from PG domain 1. [off] to [nr-0:1,R]
23:46:25:472 ra-rtr Trace: Agent 13851 from PG domain 1. [nr-0:1,R] to [rdy-0:1,RA]
23:56:56:478 ra-rtr Trace: Agent 13851 from PG domain 1. [rdy-0+1:1,R] to [res-1:1,R]
23:57:00:478 ra-rtr Trace: Agent 13851 from PG domain 1. [res-1:1,R] to [tlk-1:1,R]
• You may see (*) in Router Trace logs for agent state transitions.
This indicates a temporary state until an “ack” or timeout is
received from the agent PG (OPC) to the Router.
23:45:12:473 ra-rtr Trace: Agent 13851 from PG domain 1. [nr-0:1,R*] to [off(p)-0:0,*]
23:45:12:473 ra-rtr Trace: Agent 13851 from PG domain 1. [off(p)-0:0,*] to [off(p)-0+1:0]
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
86
UCCE Call Flow – Component Correlations
• Once the call treatment completes and/or an Agent becomes available, the
UCCE Router will ”reserve” the agent target and send a Pre-Call Indicator
alerting the Peripheral Gateway (PG) of an incoming call to the agent.
Router logs
17:02:18:576 ra-rtr Trace: (31 32 23 : 0 0) DeviceTargetPreCall_V12: CID=(152276,314), PerID=5000, SGSTID=5002, AGSTID=5001 NetworkCallFlags=0, InvokeID=141,
GREET=
PG CTISvr logs
17:02:18:594 cg1a-ctisvr Trace:
17:02:18:595 cg1a-ctisvr Trace:
17:02:18:595 cg1a-ctisvr Trace:
17:02:18:595 cg1a-ctisvr Trace:
17:02:18:595 cg1a-ctisvr Trace:
DEVICE_TARGET_PRE_CALL_IND - Instrument=1001 RouterCallKey=152276 314 1 NetworkTargetID=0
AGENT_EVENT: ID=47483648 Periph=5000 Ext=1001 Inst=1001 Sig=Finesse
SkgState=RESERVED SkgDuration=0 OverallState=RESERVED OverallDuration=0 Reason=0
MRDID=1 NumTasks=1 MaxTaskLimit=1 AgtMode=1 AGTAvailabilityStatus=0 ICMAgtID=5001
SkTgtID=5001 SkGrpNo=0x0 SkGrpID=5002 NumLines=1 CurLine=0 ClientStatus=0x1 Direction=0
Finesse logs
0001160973: 192.168.33.22: Dec 02 2017 17:02:38.872 +0100: %CCBU_CTIMessageEventExecutor-0-6-DECODED_MESSAGE_FROM_CTI_SERVER:
%[cti_message=DnD-CHECKPOINT-3B - Putting Message On Dispatcher - CTIAgentPreCallEvent [numOfNamedVariables=1, numOfNamedArrays=4,
serviceNumber=-1, serviceId=-1, skillGroupNumber=0, skillGroupId=5002, skillGroupPriority=0, mrdId=1, ani=5007, DialedNumber=2005,
RouterCallKeyDay=152276, RouterCallKeyCallID=314, RouterCallKeySequenceNum=1, callerEnteredDigits=yes, AgentID=47483648, callVariable1=null,
callVariable2=null,
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
87
UCCE Call Flow – Component Correlations
• What do all of those IDs represent???
17:02:18:576 ra-rtr Trace: (31 32 23: 0 0) DeviceTargetPreCall_V12: CID=(152276,314),
PerID=5000, SGSTID=5002, AGSTID=5001 NetworkCallFlags=0, InvokeID=141, GREET=
• “I’m not familiar with SQL queries.”
• “We’re not allowed on the database anymore after Tuesday.”
(heard on a WebEx, circa 2016)
What can you do to look up UCCE IDs?
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
88
UCCE Call Flow – Component Correlations
• There’s a tool on the Administrative Workstation (AW) - do you see it?
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
89
UCCE Call Flow – Component Correlations
• There’s a tool located in C:\icm\bin\idfinder.exe
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
90
UCCE Call Flow – Component Correlations
• The Router will send a Connect message to connect the call with the agent.
At this point, the Router is DONE.
Router logs
17:02:18:576 ra-rtr Trace: (31 32 23 : 0 0) Connect: CID=(152276,314), EventSelect=0, ServiceType=0, RCID=5002, Label(s)=1001 <--- Agent Extension
17:02:18:576 ra-rtr Trace: (31 32 23 : 0 0) Deleting Dialog.
• This connect message is passed to CVP to redirect the call from the VXML
GW to the CUCM agent device.
CVP logs
441: 192.168.33.11: Dec 02 2017 17:01:39.780 +0100: %CVP_11_5_ICM-7-CALL: {Thrd=Event-router-group-0}
CALLGUID=2679490000010000000000125636300A DialogId=31 [SIP_SWITCH_LEG_DIALOGUE] - Processing from UCCE -[CONNECT] DialogID=31
SendSeqNo=2 Label=1001 TxrHint=false rckey=152276 rcday=152276 rcseq=0 location= locationpkid= sipheader= uui= whisperAnnounce=
callguid=2679490000010000000000125636300A ECC variables=user.media.id: 2679490000010000000000125636300A, ECC
arrays=USER_MICROAPP_TOEXTVXML[0]:application=customhw, Peripheral Variables=
Establishing Ringtone-Leg
1846: 192.168.33.11: Dec 02 2017 17:01:39.796 +0100: %CVP_11_5_SIP-7-CALL: {Thrd=pool-1-thread-151-SIP-1484}
CALLGUID = 2679490000010000000000125636300A LEGID = 2679490000010000000000125636300A-151223049979634 –
[OUTBOUND]: INVITE TO <sip:91919191@192.168.33.27;transport=tcp> FROM 5007 <sip:5007@192.168.33.11:5060> EXPIRES[10] 100REL[Unsupported]
Establishing Agent-leg
1878: 192.168.33.11: Dec 02 2017 17:01:39.874 +0100: %CVP_11_5_SIP-7-CALL: {Thrd=pool-1-thread-151-SIP-1484}
CALLGUID = 2679490000010000000000125636300A LEGID = 2679490000010000000000125636300A-151223049987435 –
[OUTBOUND]: INVITE TO <sip:1001@192.168.33.20;transport=tcp> FROM 5007 <sip:5007@192.168.33.11:5060> EXPIRES[60] 100REL[Unsupported]
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
91
UCCE Call Flow – Component Correlations
CVP VXML:
Call-ID
Call GUID
VGW:
SIP Cisco-Guid
SIP Session-ID
CCAPI Call ID
CALL-ID, LEG-ID
CVP CS:
Dialog-ID
RCK, RCK Day
CALL-ID, LEG-ID
Call GUID
VVB:
Call GUID
LEG-ID
CUCM:
CI
GCH
LEG-ID
OPC:
Call GUID
RCK, RCK Day
Dialog-ID
CID
AgentPIM:
CTI Svr:
Finesse:
CID
JGW:
GCH
CI
CID
RTR:
Dialog ID
RCK, RCK Day
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
92
UCCE Email and Chat Event Flow
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
93
1.
2.
3.
Customer sends an email to the company alias.
A copy of the email is retrieved by the Services Server.
The Listener Service monitors agent states and task requests.
ECE
Message Flow
1
2
3
3
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
94
4.
5.
EAAS makes Route Request via <new task> to MR PG via MR
interface, including the Task ID and Script Selector.
Selected UCCE script is run by the Router and selects agent.
ECE
Message Flow
5
4
#CLUS
BRKCCT-2210
4
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
95
6.
7.
8.
Router sends <DO_THIS_WITH_TASK> msg to EAAS via MR
PG, including Task ID, Agent ID, and Peripheral ID.
Router sends <pre-call> msg to Agent PG, including Agent ID.
EAAS sends <OFFER_TASK> msg via ARM and assigns the
email to the selected agent’s workstation via App Server.
ECE
Message Flow
8
7
6
6
8
8
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
96
ECE
Message Flow
11. A web chat is invoked by the customer from a button on your web
site. Chat requests are handled the same way as email routing.
11
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
97
ECE Event Flow – Component Correlations
Connections and logs
Mail Server
Router
MR PG
Internet
MR pim
MR opc
•
•
Application Server
•
Firewall
Application
Server
Web Server
•
IIS Logs
•
rtr
Services Server
•
•
•
•
•
•
rx-process
dx-process
rules-process
EAAS-process
Listener-process
(11.x)
EAMS-process
(12.0)
Agent
PG
•
ctisvr
Finesse
•
Desktop
Agent
•
Fiddler/HTTPWatch
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
98
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
99
ECE Event Flow – Component Correlations
• Before we dive into the logs, there are a couple of important caveats when
troubleshooting ECE routing; especially before sending logs to TAC!
1. In UCCE 12.0, security enhancements around protecting PII (Personally
Identifiable Information) contained in serviceability logs, UCCE “hides”
Peripheral Call Variables and ECC Variables traces by default.
o This means that without enabling ECC Variable logging in UCCE, it would be near
impossible to trace an ECE event through the logs.
2. The ECE server logs are set “Error” trace levels by default.
o To net any useful information, logs MUST be set at a minimum of “Info.”
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
100
ECE Event Flow – Component Correlations
• NOTE: You cannot enable ECC Variable logging through standard tools like
RTMT, Diagnostic Framework Portico, Unified System CLI. We made it obscure
on purpose. Must use the following methods:
Router & Script Editor – rttest
To enable
debug /periph_variables
debug /ecc_variables
To disable
debug /noperiph_variables
debug /noecc_variables
PG OPC – opctest
To enable
debug /ecc_variables /periph_variables
To disable
debug /noperiph_variables (this will
disable Peripheral variable tracing)
debug /noecc_variables (this will disable
the ECC variable tracing)
PG PIMs - Procmon
trace periph_variables /[on/off]
trace ecc_variables /[on/off]
NICs & Application Gateways – Registry
EMSTraceMask = 0x8
ü Changes are dynamic
ü rttest and opctest changes are synchronous to both sides
of UCCE nodes.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
101
ECE Event Flow – Component Correlations
• Enabling tracing in ECE
o http://<ECE server>/<instance> | Log in with ‘sa’ account credentials
• NEVER set tracing above 4 unless instructed to do so by TAC. Even then, ask,
“Is this trace level safe? Will I still have a job tomorrow if I enable it?”
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
102
ECE Event Flow – Component Correlations
rx-process
IMAP/POP3
Mail Server
ECE
Messaging will vary, see mail server vendor
documentation.
2019-05-30 18:05:57.466 GMT+0000 <@> INFO <@> [87:RxInstance id :
999] <@> ProcessId:7876 <@> PID:1 <@> UID:12 <@> UserSessionId:
<@> com.egain.mail.module.retriever.service.RxProcess <@>
retrieveMailsForHostRecur<999> <@> Processing alias:
sales@uclabservices.com <@>
2019-05-30 18:05:57.765 GMT+0000 <@> INFO <@> [87:RxInstance id :
999] <@> ProcessId:7876 <@> PID:1 <@> UID:12 <@> UserSessionId:
<@> com.egain.mail.module.retriever.service.RxProcess <@>
retrieveMailsForHostRecur<999> <@> Number of Messages: 1 <@>
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
103
ECE Event Flow – Component Correlations
rx-process
ECE
rules-process
ECE
2019-05-30 18:06:13.489 GMT+0000 <@> INFO <@> [94:BPRulesInstance id : 996] <@> ProcessId:10896 <@> PID:1 <@> UID:12 <@> UserSessionId: <@>
com.egain.platform.module.routing.service.bprules.BPRulesInstance <@> doWork() <@> Processing - Activity Id: 1017 <@>
2019-05-30 18:06:13.498 GMT+0000 <@> INFO <@> [94:BPRulesInstance id : 996] <@> ProcessId:10896 <@> PID:1 <@> UID:12 <@> UserSessionId: <@>
com.egain.platform.module.routing.service.DefaultEngine <@> applyWorkflow(CallerContext oCallerContext, RuleContext oContext, Workflow oWorkflow, long
lCurrentItemId) <@> Executing Workflow: Start Workflow - Standard - Activity Id: 1017 <@>
2019-05-30 18:06:13.602 GMT+0000 <@> INFO <@> [94:BPRulesInstance id : 996] <@> ProcessId:10896 <@> PID:1 <@> UID:12 <@> UserSessionId: <@>
com.egain.platform.module.routing.service.DefaultEngine <@> applyWorkflow(CallerContext oCallerContext, RuleContext oContext, Workflow oWorkflow, long
lCurrentItemId) <@> Executing Workflow: Start Workflow - Sales QueueMessages - Activity Id: 1017 <@>
2019-05-30 18:06:13.602 GMT+0000 <@> INFO <@> [94:BPRulesInstance id : 996] <@> ProcessId:10896 <@> PID:1 <@> UID:12 <@> UserSessionId: <@>
com.egain.platform.module.routing.service.DefaultEngine <@> applyWorkflow(CallerContext oCallerContext, RuleContext oContext, Workflow oWorkflow, long
lCurrentItemId) <@> Executing Workflow item: Service Level 2 - Activity Id: 1017 <@>
2019-05-30 18:06:13.621 GMT+0000 <@> INFO <@> [94:BPRulesInstance id : 996] <@> ProcessId:10896 <@> PID:1 <@> UID:12 <@> UserSessionId: <@>
com.egain.platform.module.routing.service.bprules.RuleContextMail <@> routeActivity() <@> Activity moved to Queue Id: 1007 - Activity Id: 1017 - approval_type
configured for the queue: <@>
ECE rules-process logs
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
104
ECE Event Flow – Component Correlations
• All ECE activities have the following:
o CaseID – ID of interaction between company and customer.
o ActivityID – ID for each task or item.
• There is a 1 to many relationship between the CaseID and ActivityID.
o In the workflow example below, there’s 1 CaseID and 2 ActivityIDs for an inbound email.
o The first ActivityID is the inbound email itself | The second ActivityID is the Auto Acknowledgement
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
105
ECE Event Flow – Component Correlations
EAAS-process
ECE
2019-05-30 18:06:18.619 GMT+0000 <@> INFO <@> [94:Thread-11] <@> ProcessId:5384 <@> PID:1 <@> UID:12 <@>
UserSessionId: <@> com.ipcc.mr.NonRealtimeActivityHandler <@> fetchActivities() <@> list : 1 Collection containing 1
elements: [[Property = ActivityId & value = 1017
Property = CaseId & value = 1008
Property = DepartmentId & value = 999
Property = ActivityType & value = 1
Property = ActivityMode & value = 0
…
{call_variable_custom_attribute=activity_id, task_group_low=-1, sequence_number=-1, reroute_flag=0,
call_and_ecc_var=[call_variable_custom_attribute = activity_id : Type = java.lang.String
call_variable_name = activity_id : Type = java.lang.String
call_variable_object = casemgmt::activity_data : Type = java.lang.String
call_variable_tag = null
ecc_variable_id = 1 : Type = java.math.BigDecimal
script_selector_id = 5002 : Type = java.math.BigDecimal
], task_group_high=-1, call_variable_name=activity_id, queue_priority=0.00, call_variable_object=casemgmt::activity_data,
ecc_variable_id=1, mrd_id=5001, script_selector_id=5002}]] <@>
2019-05-30 18:06:18.671 GMT+0000 <@> INFO <@> [94:Thread-11] <@> ProcessId:5384 <@> PID:1 <@> UID:12 <@>
UserSessionId: <@> com.ipcc.mr.ActivityHandlerQueue <@> <@> Exiting incrementCount -> mrdId = 5001, sdo ->
max_task_limit = 10000 : Type = java.lang.Integer
queued_task = 1 : Type = java.lang.Integer
<@>
2019-05-30 18:06:18.673 GMT+0000 <@> INFO <@> [21200:mr-request-executor::-1] <@> ProcessId:5384 <@> PID:1 <@>
UID:12 <@> UserSessionId: <@> com.ipcc.arm.MRLogger <@> logOutgoingMessage() <@> MSG_TYP_NEW_TASK -> 0 0 0
93 0 0 4 76 0 0 0 2 0 0 0 1 0 0 19 -119 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 99 15 69
77 65 73 76 95 83 65 76 69 83 95 83 83 0 82 26 117 115 101 114 46 101 99 101 46 97 99 116 105 118 105 116 121 46 105 100 0
49 48 49 55 0 <@>
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
106
ECE Event Flow – Component Correlations
MR
PIM
EAAS
ECE
14:06:18:636 PG3A-pim1 Trace: Application->PG:
Message = NEW_TASK; Length = 93 bytes
DialogueID = (2) Hex 00000002
SendSeqNo = (1) Hex 00000001
MRDomainID = (5001) Hex 00001389
PreviousTask = -1:-1:-1
PreferredAgent = Undefined
Service = (0) Hex 00000000
CiscoReserved = (0) Hex 00000000
ScriptSelector: EMAIL_SALES_SS
ServiceRequested = (0) Hex 00000000
QueueType = (0) Hex 00000000
QueueID = (0) Hex 00000000
2019-05-30 18:06:18.673 GMT+0000 <@> INFO
<@> [21200:mr-request-executor::-1] <@>
ProcessId:5384 <@> PID:1 <@> UID:12 <@>
UserSessionId: <@>
com.ipcc.mr.RequestExecutorCmd <@>
RequestExecutorCmd() <@> New task message
processing completed with Status=truefor
activityId=1017. EAASInstanceId=888 <@>
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
107
ECE Event Flow – Component Correlations
MR PIM
Pre-12.0
12.0
14:06:18:636 PG3A-pim1 Trace: Application->PG:
Message = NEW_TASK; Length = 93 bytes
DialogueID = (2) Hex 00000002
SendSeqNo = (1) Hex 00000001
MRDomainID = (5001) Hex 00001389
PreviousTask = -1:-1:-1
PreferredAgent = Undefined
Service = (0) Hex 00000000
CiscoReserved = (0) Hex 00000000
ScriptSelector: EMAIL_SALES_SS
ServiceRequested = (0) Hex 00000000
QueueType = (0) Hex 00000000
QueueID = (0) Hex 00000000
14:06:18:636 PG3A-pim1 Trace: Application->PG:
Message = NEW_TASK; Length = 131 bytes
DialogueID = (2) Hex 00000002
SendSeqNo = (1) Hex 00000001
MRDomainID = (5001) Hex 00001389
PreviousTask = -1:-1:-1
PreferredAgent = Undefined
Service = (0) Hex 00000000
CiscoReserved = (0) Hex 00000000
ScriptSelector: EMAIL_SALES_SS
ServiceRequested = (0) Hex 00000000
QueueType = (0) Hex 00000000
QueueID = (0) Hex 00000000
Call Variable 1:
ECC Variable Name: user.ece.activity.id
Value: 1017
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
108
ECE Event Flow – Component Correlations
MR
PIM
14:06:18:636 PG3A-pim1 Trace: Application->PG:
Message = NEW_TASK; Length = 93 bytes
DialogueID = (2) Hex 00000002
SendSeqNo = (1) Hex 00000001
MRDomainID = (5001) Hex 00001389
PreviousTask = -1:-1:-1
PreferredAgent = Undefined
Service = (0) Hex 00000000
CiscoReserved = (0) Hex 00000000
ScriptSelector: EMAIL_SALES_SS
ServiceRequested = (0) Hex 00000000
QueueType = (0) Hex 00000000
QueueID = (0) Hex 00000000
ECC Variable Name: user.ece.activity.id
Value: 1017
MR
OPC
14:06:18:651 PG3A-opc Trace: Call:SetCallState (PID=5003) CallID=2(0x15900d8)[SrcDev() DestDev()] NewState=NULL
OldState=NONE.
14:06:18:651 PG3A-opc Trace: SendINRCMessage: NEW_CALL
RCID=5003 PID=5002 DID=2 DIDRelSeq#=1 GrpDelKey=5003
CRS(RtrDate=-1,RtrCID=-1) RCKSeq#=-1 NICCallID={N/A}
PGCallID={N/A} OperationCode=CLASSIC OperationFlags=COOP_NONE
CalledParty#=EMAIL_SALES_SS CallingParty#= CED= MsgOrigin=1
RouteDevType=6 Originator=cclab12ecesv NICCalledParty#= NetDelay=0
Priority=0 OrigType=0 atVRU=T CLIDRestrict=F PrefAGSTID=-1
MRDomainID=5001 LegacyPreCallMsgEnabled=F CID=0, ServiceReq=0
QueueType=0 QueueID=0 PV1= PV2= PV3= PV4= PV5= PV6= PV7=
PV8= PV9= PV10=
Name
Type
Index Value
user.ece.activity.id
Scalar
1017
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
109
ECE Event Flow – Component Correlations
MR
OPC
Router
rtr
14:06:18:651 PG3A-opc Trace: Call:SetCallState (PID=5003) CallID=2(0x15900d8)[SrcDev() DestDev()] NewState=NULL
OldState=NONE.
14:06:18:330 ra-rtr Trace: (2 x 0 : 0 0) NewCall: CID=(152820,401),
DN=EMAIL_SALES_SS, ANI=, CED=, RCID=5003, MRDID=5001,
CallAtVRU=1, OpCode=0
14:06:18:651 PG3A-opc Trace: SendINRCMessage: NEW_CALL
RCID=5003 PID=5002 DID=2 DIDRelSeq#=1 GrpDelKey=5003
CRS(RtrDate=-1,RtrCID=-1) RCKSeq#=-1 NICCallID={N/A}
PGCallID={N/A} OperationCode=CLASSIC OperationFlags=COOP_NONE
CalledParty#=EMAIL_SALES_SS CallingParty#= CED= MsgOrigin=1
RouteDevType=6 Originator=cclab12ecesv NICCalledParty#= NetDelay=0
Priority=0 OrigType=0 atVRU=T CLIDRestrict=F PrefAGSTID=-1
MRDomainID=5001 LegacyPreCallMsgEnabled=F CID=0, ServiceReq=0
QueueType=0 QueueID=0 PV1= PV2= PV3= PV4= PV5= PV6= PV7=
PV8= PV9= PV10=
Name
Type
Index Value
user.ece.activity.id
Scalar
1017
23:14:33:424 ra-rtr Trace: Script-Execute CID=(152820,401)
Default\\UCLAB_ECE_SALES_Q Start 8
23:14:33:424 rb-rtr Trace: Script-Execute CID=(152820,401)
Default\\UCLAB_ECE_SALES_Q Queue Agent
14:06:18:330 ra-rtr Trace: 5000
#CLUS
BRKCCT-2210
user.ece.activity.id
Scalar
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
1017
110
ECE Event Flow – Component Correlations
Router
rtr
16:22:00:266 ra-rtr Trace: (2 x 0 : 0 0) RouteComplete:
16:22:00:266 ra-rtr Trace: Route: DN=EMAIL_SALES_SS, ANI=, CED=, Label=
16:22:00:266 ra-rtr Trace: Route: CID=(152820,401), Labels=1
16:22:00:266 ra-rtr Trace: (2 x 0 : 0 0) DeviceTargetPreCall_V13:
CID=(152820,401), PerID=5000, SGSTID=5019, AGSTID=5014
NetworkCallFlags=0, InvokeID=15, GREET=
16:22:00:266 ra-rtr Trace: DN=EMAIL_SALES_SS, ANI=, CED=,
CustDBProvidedDigits=, RouteID=5002, NetworkTargetID=0
16:22:00:266 ra-rtr Trace: RCKSeqNum=0, DelayQTime=8141,
DelayRouterTime=8141, SGDelayQTime=8141
16:22:00:266 ra-rtr Trace: CallGUID=, CustomerID=-1 ServiceRequested=0
16:22:00:266 ra-rtr Trace: LocationParamPKID=, LocationParamName=,
SIPHeader=
16:22:00:266 ra-rtr Trace: NIC_CalledPartyNumber=EMAIL_SALES_SS
ECCPayloadID=1
16:22:00:266 ra-rtr Trace: PV1= PV2= PV3= PV4= PV5=
16:22:00:266 ra-rtr Trace: PV6= PV7= PV8= PV9= PV10=
16:22:00:266 ra-rtr Trace: ID
Name
Type Index Value
16:22:00:266 ra-rtr Trace: 5000 user.ece.activity.id
Scalar
1017
MR
OPC
16:22:00:352 PG3A-opc Trace: SendPIMINRCMessage: CONNECT
RCID=5003 PID=5002 DID=2 DIDRelSeq#=0
CRS(RtrDate=152820,RtrCID=401) RCKSeq#=0 ErrorCode=0 TRTargetID=1 CorrID= EventSel=0 SvcType=0 NICCallID={N/A} PGCallID={N/A}
OperationCode=CLASSIC OperationFlags=COOP_NONE
NetworkTransferEnabled=F ECCPayloadID=5002
NICCalledParty#=EMAIL_SALES_SS SGSTID=5019 PQID=0 SvcSTID=-1
AGSTID=5014 AGInfo=1012, MRDID=5001 Interruptible=1 PV1= PV2=
PV3= PV4= PV5= PV6= PV7= PV8= PV9= PV10=
Name
Type
Index Value
user.ece.activity.id
Scalar
1017
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
111
ECE Event Flow – Component Correlations
EAAS
MR
PIM
ECE
16:22:00:352 PG3A-pim1 Trace: PG->Application:
Message = DO_THIS_WITH_TASK; Length = 102 bytes
DialogueID = (2) Hex 00000002
SendSeqNo = (1) Hex 00000001
IcmTaskID = 152820:401: 1
SkillGroup = (5019) Hex 0000139b
Service = Undefined
Agent = (5014) Hex 00001396
AgentInfo: 1012
Label:
ApplicationString2:
MRDID = (5001) Hex 00001389
Interruptible = (1) Hex 00000001
PrecisionQueueID = (0) Hex 00000000
#CLUS
2019-05-30 20:22:00.782 GMT+0000 <@> INFO <@> [21528:mrresponse-executor-executor::-1] <@> ProcessId:5384 <@> PID:1 <@>
UID:12 <@> UserSessionId: <@> com.ipcc.mr.ResponseExecutorCmd
<@> <@> DO_THIS_WITH_TASK Processing complete. in EAAS
EAASinstanceId=888 processed message:do_this_with_task =
DoThisWithTaskMsg [mDialogId=2, mSendSeqNum=1,
mICMTaskId=152820_401_1, mICMId=null, mSkillGroup=5019,
mService=-1, mAgent=5014, mMRDId=5001, mIsInterruptible=true,
mPrecisionQueueId=0, mAgentInfo=1012, mLabel=, mAppStr2=,
mScriptSelector=EMAIL_SALES_SS, mAppStr1=, mSubType=1] : Type =
com.ipcc.mr.message.DoThisWithTaskMsg
entry_point_id = 0 : Type = java.lang.Long
do_this_with_task_type = 0 : Type = java.lang.Integer
expired_status = false : Type = java.lang.Boolean
user_id = 1012 : Type = java.lang.Long
customer_home = null
true = true : Type = java.lang.String
activity_id = 1017 : Type = java.lang.Long
activity_sub_type = 1 : Type = java.lang.Integer
mrd_id = 5001 : Type = java.lang.Long
<@>
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
112
ECE Event Flow – Component Correlations
• Relevant logs to capture
ECE logs
Location: %Home\eService\logs
eg_log_<Server Name>_EAAS-process.log
eg_log_<Server Name>_Listener-process.log
eg_log_<Server Name>_rx-process.log
eg_log_<Server Name>_Application Server.log
UCCE logs
MediaRouting (PG) PIM
Router (rtr)
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
113
UCCE Finesse
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
114
Finesse Client ó Server Relationship
• Finesse clients connect to Primary / Secondary active | active servers.
• Critical latency benchmarks:
o <400ms RTT between Client and Server (best practice)
o >30 seconds will lead to Agents/Supervisors complaining of a Finesse “red bar”
o >60 seconds may result in Agents/Supervisors getting logged out.
Agent or Supervisor Desktop
PUT
http://finessea.cc.lab/finesse/api/User/47483648
<User>
<state>NOT_READY</state>
<reasonCodeId>1</reasonCodeId>
</User>
HTTP REST API
PG CTISvr
Finesse
VOS
XMPP Events
#CLUS
GED-188 protocol
BOSH tunnel
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
115
Finesse Client ó Server Relationship
• The BOSH tunnel is set up between the Agent and Finesse server every 30
seconds. The XMPP messages flow through this tunnel.
• You will see this in the http-bind messages where status “200” means that the
Finesse server has acknowledged the successful session setup with the client.
Finesse
XMPP Events
#CLUS
BRKCCT-2210
BOSH tunnel
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
116
Finesse Client ó Server Relationship
• Agents and Supervisors will see a temporary “red bar/banner” when there’s an
issue with the XMPP BOSH session.
• Condition can last between 1 to 30 seconds.
o Self-correcting and no manual recovery needed if under 30 seconds.
o During this time, while CTI buttons are enabled, agents will not have CTI controls
• If the condition lasts for more than 30 seconds, you run the risk of Finesse
marking the agent as not routable and/or logging the agent out.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
117
Finesse Client ó Server Relationship
• Example: Lets say that 10 seconds in to the BOSH session there’s an issue.
The agent or supervisor won’t see the “red bar” until another 20 seconds. This
is because the BOSH tunnels are set up every 30 seconds. During this time, if
the agent is active and invokes a CTI call (make call, not ready, etc.), the
agent/supervisor will see the following:
• After about 20 seconds, the agent will receive the “red bar” message.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
118
Finesse Client ó Server Relationship
• Troubleshooting a Finesse fault requires the following (core) tools:
1. The use of the Finesse client’s “Send Error Report” button
Send Browser Console log
to the Finesse server
Open Finesse logs in browser: https://Finesse_FQDN:8445/finesse/logs/clientlogs/
Agent peripheral ID
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
119
Finesse Client ó Server Relationship
• Troubleshooting a Finesse fault requires the following (core) tools:
2. Enabling the client browser’s console logs by pressing F12
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
120
Finesse Client ó Server Relationship
• Troubleshooting a Finesse fault requires the following (core) tools:
3. Collecting the Finesse server logs
WEB
https://Finesse_FQDN:8445/finesse/logs/
RTMT
CLI
file get activelog /desktop/logs/* recurs
compress
Finesse WebServices Log contains interaction with CTI server and Finesse Clients
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
121
Finesse Client ó Server Relationship
• The Finesse “Send Error Report” has a limitation.
• Sometimes you are troubleshooting an issue where the client failure is a
browser crash, freeze, and/or timing prevents an agent from being able to
press the Send Error Report button.
• This is when Persistent Logging can help. Log into Finesse normally but using
this URL to enable persistent logging: https://finesse_server/desktop/locallog
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
122
Finesse Client ó Server Relationship
• Once you have captured the event/condition, simply copy/paste the log events
into a text file.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
123
Finesse Client ó Server Relationship
• If the next BOSH session is successfully established, the agent/supervisor will
re-gain CTI controls and the “red bar” will go away.
• There are a lot of possible conditions that could trigger intermittent BOSH
session failures so it’s important to understand this integration behavior.
• You will need to enable the browser console logs (press F12) capture those
logs with the Finesse server logs for further analysis.
• Extremely deep-dive troubleshooting Finesse (BOSH) disconnects:
https://www.cisco.com/c/en/us/support/docs/customer-collaboration/unifiedcontact-center-express-1151/211376-Technote-on-How-Bosh-ConnectionWorks-fo.html
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
124
Finesse Client Agent State Changes
• Another common condition is an Agent logout events.
select * from t_Agent_Logout
Predefined Reason Codes:
50002 - This could be due to closing the agent desktop application, heartbeat timeout, or a CTI server client failure (such as Finesse).
50003 - The agent was logged out because the Unified CM reported the agent's device as out of service.
50004 - The agent was logged out due to agent inactivity as configured in agent desk settings.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
125
Finesse Client Agent State Changes
• A quick and simple way to view agent changes.
1 = LOGIN
2 = LOGOUT
3 = NOT_READY
select * from t_Agent_Event_Detail
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
126
Leveraging the
Database
UCCE’s t_Event Table – A Hidden Gem!
• How often do you leverage UCCE “t_Event” table?
• UCCE’s t_Event table is an aggregation of the application’s events which
include SNMP, syslog, and Router Log Viewer.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
128
UCCE’s t_Event Table – A Hidden Gem!
• Found on UCCE Loggers | 14-day retention by default
• select * from t_Event where CentralControllerFileTime between
'YYYY-MM-DD 00:00:00' and 'YYYY-MM-DD 23:59:59'
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
129
UCCE’s t_Event Table – A Hidden Gem!
• Export the SQL query as .CSV with headers.
• Create filters:
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
130
UCCE’s DumpCFG – Another Hidden Gem!
• The “Trust but verify” UCCE (command line) utility.
• The main function of the dumpcfg tool is to dump the Recovery Key in the
Config_Message_Log table of the Logger database.
• The Config_Message_Log table keeps track of all configuration changes in
UCCE.
o Any and all script changes and updates
o Agent and Supervisor add/delete/modify
o Routing client updates
• Technical Tip: https://www.cisco.com/c/en/us/support/docs/voice-unifiedcommunications/icm-logger/69985-dumpcfg-icmconfig.html
o The tech-tip will still work but there has been an UPDATE to the dumpcf utility.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
131
UCCE’s DumpCFG – Another Hidden Gem!
• Historically, you had to run a SQL query to extrapolate the Max
and Min Recovery Keys from the Config_Messag_Log table on
the UCCE Logger:
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
132
UCCE’s DumpCFG – Another Hidden Gem!
• Next, you would have to apply the Max and Min values as
shown here:
Syntax
dumpcfg <database>@<server> <low recovery key> <high recovery key>
dumpcfg cisco_sideA@10.77.56.3 517558521000 517558521003
• However, dumpcfg has learned a few
tricks:
o You no longer need to run the SQL query to get the max and min RecoveryKey values
o Dumpcfg allows you to directly input date and time and day ranges.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
133
UCCE’s DumpCFG – Another Hidden Gem!
[ 517558521000.0, Add, Jan 1 2019 08:26AM ]
PID:
4980
MachineName:
IPCCLABAW01
UserDomain:
IPCCLAB
UserName:
mikchen
ProcessName:
trwizard(upcc)
SQLServerUserName: dbo
[ 517558521001.0, Delete, t_Translation_Route, Jan 1 2019 08:26AM ]
SkillTargetID:
35287
ChangeStamp:
0
[ 517558521002.0, Add, Jan 1 2019 10:05AM ]
PID:
4848
MachineName:
IPCCLABAW01
UserDomain:
IPCCLAB
UserName:
mikchen
ProcessName:
scripted(upcc)
SQLServerUserName: DBO
[517558521002.0, Update, t_Master_Script, Jan 1 2019 10:05AM ]
ObjectAccessLevel: 0
MasterScriptID: 5067
CustomerDefinitionID: 0
CustomerIdShadow: 0
EntityID:
1
CurrentVersion: 4294967295
NextAvailableVersion: 178
ChangeStamp:
327
ScriptType:
1
EnterpriseName: PROMPTER
Description:
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
134
UCCE’s DumpCFG – Another Hidden Gem!
• You can also capture the dumpcfg file via UCCE Portico:
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
135
Summary
• Troubleshooting can be complicated, frustrating, exciting, and fun!
• Malcolm Gladwell’s 10,000-hours rule: https://en.wikipedia.org/wiki/Outliers_(book)
• You now have the
to troubleshoot UCCE. Ready. Aim. Fire!
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
136
Complete your
online session
evaluation
•
Please complete your session survey
after each session. Your feedback
is very important.
•
Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
•
All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
137
Upgrade /
Migration
BRKCCT-1051
Mon 1-3 pm
BRKCCT-2007
Wed 4-5:30 pm
BRKCCT-2210
Thu 8-10 am
BRKCCT-2001
Thu 1-2:30PM
BRKCCT-1011
Mon 1-2:30 pm
BRKCCT-1511
Mon 4-5:30 pm
Techtorials
Techtorial
Techtorial
UCCE
TECCT-2022
Sun 9 am -1 pm
TECCT-3002
Sun 9 am- 6 pm
TECCT-2005
Sun 9 am–1 pm
BRKCCT-2035
Wed 8-10 am
Analytics
Desktop
AI/BOTs
Instructor led
labs:
Intermediate
Instructor led
labs:
Advanced
DevNet Sessions
UCCE
UCCX
LTRCCT-2051
Mon 8 am-noon
LTRCCT-2010
Tue 1-5 pm
UCCE
CJP
LTRCCT-3052
Mon 1-5 pm
LTRCCT-1050
Sun 2-6 pm
& 4 Walk-in labs
#CLUS
Integration
BRKCCT-2004
Tue 8-10 am
BRKCCT-2004
Tue 8-10 am
Instructor led hands-on
CVP
UCCX
Security
BRKCCT-2009
Wed 4-5:30 pm
Troubleshooting
BRKCCT2010
Tue 30:305:30 pm
Design &
Planning
BRKCCT-1003
Tue 1-2 pm
CJP
BRKCCT-1002
Tuesday 4-5 pm
UCC
X
PSOCCT-1645
Monday 8-9 am
UCCE
Roadmap /
Features
BRKCCT-1001
Thu 10:30-noon
Flex
BRKCCT-2002
Tue 1-3 pm
Product
strategy
BRKCCT-2056
Wed 1-3 pm
Continue your customer journey solutions education
ECE
Omni-channel
DEVWKS-1489
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
138
Continue your education
Demos in the
Cisco campus
Walk-in
self-paced labs
Meet the engineer
1:1 meetings
Related sessions
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
139
Thank you
#CLUS
#CLUS
Appendix A
Session ID
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
142
Data Monitoring and Telemetry Capabilities
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
For 12.0
143
Data Monitoring and Telemetry Capabilities
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
For 12.0
144
Data Monitoring and Telemetry Capabilities
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
For 12.0
145
Data Monitoring and Telemetry Capabilities
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
For 12.0
146
Data Monitoring and Telemetry Capabilities
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
For 12.0
147
Data Monitoring and Telemetry Capabilities
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
For 12.0
148
UCCE Comprehensive Call Flow
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
149
UCCE Call Flow
1.
PSTN
1
1
2.
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Invite
SIP Proxy
2
CUSP
SIP
SIP Invite
SIP
The call arrives at the CUBE SBC or TDM-IP Gateway, also
commonly called the Ingress Gateway.
a. The Ingress Gateway converts TDM to VoIP.
b. It also assigns a 32-digit hexadecimal Global Unique
Identifier (GUID) to the call.
The Ingress Gateway then sends a SIP Invite to the SIP Proxy
Server.
a. The SIP Proxy Server determines the IP address of the
CVP Call Server for the dialed number.
b. The SIP Proxy Server forwards the SIP Invite from the
Ingress Gateway to the selected CVP Call Server’s SIP
Service.
c. This establishes the Switch Leg of the call.
2
VXML
Gateway
SIP Service
CUCM
CUCM
Subscriber
Subscriber
Agent PG
ICM Service
EAPIM
Switch Leg
JTAPI GW
JTAPI
Client
CTI Manager
JTAPI
UCCE
VRU PG
Router
VRU PIMs
CVP Call Server
VXML over HTTP
GED-125 (SCI)
DMP
IVR Service
Media
Server
VXML Server
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
150
UCCE Call Flow
3.
4.
5.
PSTN
The SIP Service sends a New Call request to the ICM Service.
The ICM Service sends the New Call request to the VRU PG.
This request causes ICM to run a routing script based upon the
dialed number and other criteria.
a. To provide self-service, the ICM routing script utilizes a
Send to VRU node to return a label to the VRU Routing
Client to have the call sent to a VoiceXML gateway.
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
SIP
SIP
VXML
Gateway
SIP Service
3
Switch Leg
EAPIM
New Call
JTAPI GW
4
CVP Call Server
VXML over HTTP
CUCM
CUCM
Subscriber
Subscriber
Agent PG
ICM Service
JTAPI
Client
CTI Manager
JTAPI
UCCE
VRU PG
Router
VRU PIMs
New Call
5
GED-125 (SCI)
DMP
IVR Service
Media
Server
VXML Server
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
151
UCCE Call Flow
6.
7.
ICM returns a Temp-connect message to the Call Server.
ICM then immediately returns a Connect instruction to the Call
Server along with a VRU Label (Assume the VRU Label is 10
digits) with a unique 5-digit Correlation ID.
8. The ICM Service sends this Connect instruction with the 15-digit
VRU Label + Correlation ID to the SIP Service.
9. The SIP Service sends a request to the SIP Proxy Server to find
the IP address of the VXML Gateway associated with the label
returned by ICM.
10. The SIP Service sends a SIP Invite to the VXML Gateway and
waits for the OK response.
PSTN
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
SIP
SIP
9
VXML
Gateway
10
SIP Service
Switch Leg
EAPIM
8
JTAPI GW
Connect
SIPOK
Invite
CUCM
CUCM
Subscriber
Subscriber
Agent PG
ICM Service
7
CTI Manager
JTAPI
VRU PG
6
CVP Call Server
JTAPI
Client
VRU PIMs
UCCE
6
Router
Temp-connect
Connect
VXML over HTTP
VXML over HTTP
7
GED-125 (SCI)
DMP
IVR Service
Media
Server
VXML Server
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
152
UCCE Call Flow
11. The SIP Service sends an instruction to the Ingress Gateway to
direct the RTP audio stream to the VXML Gateway.
12. The Ingress Gateway sends the audio as a New Call to the
VXML Gateway using the VRU Label + Correlation ID as the
Dialed Number.
13. The VXML Gateway must have a dial peer configured to match
the VRU Label—plus a wild card to allow for the extra digits of
the Correlation ID. This dial peer executes the bootstrap.tcl
script that extracts the Call Server IP and starts the VXML
Interpreter (voice browser) for this call.
14. The VXML Gateway makes a New Call request using the VRU
Label + Correlation ID as the Dialed Number. This connects with
the IVR Service to establish the VRU Leg of the call.
PSTN
TDM/SIP
New Call
CUBE/TDM-IP
to
Gateway
VRU Label
+
Correlation ID
SIP Proxy
CUSP
SIP
SIP
12
11
CUCM
CUCM
Subscriber
Subscriber
RTP Stream
VXML 13
Gateway
SIP Service
Agent PG
ICM Service
EAPIM
Switch Leg
JTAPI GW
JTAPI
Client
CTI Manager
JTAPI
UCCE
VRU PG
New Call
Router
VRU PIMs
CVP Call Server
VXML over HTTP
VXML over HTTP
GED-125 (SCI)
14
DMP
IVR Service
VRU Leg
VXML Server
Media
Server
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
153
UCCE Call Flow
15. The IVR Service strips off the Correlation ID and converts this
into a “request instruction” message for the call identified by the
Correlation ID. This is sent to the ICM Service.
16. The ICM Service passes this along to ICM.
17. ICM continues the routing script of the original call, continuing
down the SUCCESS path of the Send to VRU node.
18. The ICM routing script then executes a Run External Script node
to execute a micro-application to provide VXML Self-Service to
the caller. The “GS,Server,V” (GetSubdialog) script is required
for CVP Call Studio self-service.
19. ICM returns the instruction to run the micro-application and the
configured variables to the ICM Service.
20. The ICM Service passes this to the IVR Service.
PSTN
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
SIP
SIP
CUCM
CUCM
Subscriber
Subscriber
RTP Stream
VXML
Gateway
SIP Service
Agent PG
ICM Service
EAPIM
Switch Leg
JTAPI GW
Run
Instruction
CVP Call Server
VXML over HTTP
VXML over HTTP
VXML Server
Request
Instruction
CTI Manager
JTAPI
19
VRU PG
VRU PIMs
16
16
Run
UCCE
Instruction
Router
GED-125 (SCI)
DMP
15
IVR Service
Request
VRU Leg
Instruction
20
19
JTAPI
Client
18
Media
Server
ASR
Server
17
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
154
UCCE Call Flow
21. All micro-applications cause the IVR Service to find a template
of VXML code on the Call Server and insert the ICM data to
configure that template. For GS,Server,V the Call Server uses
the GetSpeech-External.template.
22. This VXML code is returned in the http response stream to the
VXML Gateway.
23. The VXML Gateway pushes the current information onto a stack
and executes the Subdialog statement to invoke the VXML
Server application.
24. The VXML Server starts a New Session for this caller and
begins executing the Studio application script.
25. Each interaction with the caller sends a separate VXML page to
the VXML Gateway.
PSTN
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
SIP
SIP
CUCM
Subscriber
RTP Stream
VXML
Gateway
SIP Service
Agent PG
ICM Service
EAPIM
Switch Leg
JTAPI GW
JTAPI
Client
CTI Manager
JTAPI
UCCE
VRU PG
22
VXML over HTTP
21
VXML over HTTP
GED-125 (SCI)
25
24
Router
VRU PIMs
CVP Call Server
VXML Code
DMP
23
IVRVXML
Service
Page
New
Session
VRU
Leg
VXML Server
Media
Server
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
155
UCCE Call Flow
PSTN
Please enter your 4-digit
account number.
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
26. The VXML Gateway tries to retrieve pre-recorded audio from its
cache. If not found, or if expired, the VXML Gateway retrieves it
from the Media Server, caches the audio file, and plays it to the
caller.
27. After each interaction with the caller, the VXML Gateway
submits data back to the VXML Server for logging purposes and
to continue the application’s call flow.
28. The VXML Server continues the application.
29. In this case, the application requests 4-digit input from the caller,
using ASR or DTMF.
CUSP
SIP
SIP
CUCM
Subscriber
RTP Stream
26
VXML
Gateway
SIP Service
Agent PG
ICM Service
EAPIM
Switch Leg
JTAPI GW
Cache
JTAPI
Client
CTI Manager
JTAPI
UCCE
VRU PG
Router
VRU PIMs
CVP Call Server
Logging Data
VXML over HTTP
VXML over HTTP
27
GED-125 (SCI)
29
DMP
26
IVR Service
28
VRU Leg
getAcctNum
VXML Server
Media
Server
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
156
UCCE Call Flow
30. This is carried out by the VXML Gateway. It uses MRCP to
instruct the ASR Server to load the Digits grammar and start
listening.
The VXML Gateway also checks the cache for the audio
prompt or retrieves it from the Media Server, if necessary.
The VXML Gateway plays the outgoing prompt and sends all
audio input to the ASR Server.
31. Information is always returned from the VXML Gateway to the
VXML Server for logging purposes and to continue the
application’s call flow.
32. This continues until the Studio application executes a CVP
Subdialog Return element. Five variables can be returned to
ICM with data: Caller_Input and FromExtVXML0, 1, 2, 3.
PSTN
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
SIP
SIP
CUCM
Subscriber
RTP Stream
30
VXML
Gateway
<vxml>
<form>
VXML over HTTP
31
EAPIM
Switch Leg
JTAPI GW
CTI Manager
JTAPI
UCCE
VRU PG
Router
VRU PIMs
CVP Call Server
</form>
</vxml>
JTAPI
Client
VXML over HTTP
GED-125 (SCI)
DMP
30
32
Agent PG
ICM Service
<block>
<return namelist=“caller_input FromExtVXML0
FromExtVXML1 FromExtVXML2 FromExtVXML3” />
</block>
Cache
Load
“Digits”Data
Grammar
Logging
Start Listening
SIP Service
30
IVR Service
VRU LegReturn
CVP Subdialog
VXML Server
Media
Server
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
157
UCCE Call Flow
PSTN
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
SIP
SIP
RTP Stream
VXML
Gateway
SIP Service
ICM Service
33. The CVP Subdialog Return causes the VXML Server to send a
page to the VXML Gateway, returning from the existing
‘Subdialog’ with the data returned back.
34. The VXML Gateway returns data from the VXML Server to the
IVR Service.
35. The data and a result=success is sent from the IVR Service to
the ICM Service.
36. Data is relayed from the ICM Service to ICM into the
Caller_Input variable and (optionally) into the FromExtVXML[0],
[1], [2], [3] array.
The ICM routing script continues down the SUCCESS path of
the Run External Script (GS,Server,V) node.
37. After analyzing the returned data, the routing script might
change the call type and queue the call to a skill group or
precision routing group.
CUCM
Subscriber
Agent PG
EAPIM
Switch Leg
JTAPI GW
CVP Call Server
VXML over HTTP
VXML over HTTP
IVR
Service
caller_input,
VXML
FromExtVxml0,1,2,3
Page
VRU Leg
result=success
VXML Server
Media
Server
36
Router
VRU PIMs
caller_input,
FromExtVxml0,1,2,3
UCCE
GED-125 (SCI)
35
33 34
CTI Manager
JTAPI
VRU PG
36
http submit: caller_input,
FromExtVxml0,1,2,3
JTAPI
Client
DMP
37
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
158
UCCE Call Flow
38. While the caller is in queue, ICM must provide music to the caller
using another microapp, such as the PlayMedia (PM,
<audioFileName>). Note that you could send the call to another
Studio application to play audio files until ICM interrupts and
sends the caller to an agent.
39. This example shows ICM sending the request for the PlayMedia
(PM,HoldMusic,S) microapp.
40. The request is sent to the IVR Service, which opens the
PlayMedia template file and configures it based on data received
from the routing script.
41. The VXML page is sent to the VXML Gateway to play the audio
file.
42. The VXML Gateway plays the audio file from cache or retrieves
it from the Media Server, if necessary.
PSTN
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
SIP
SIP
CUCM
Subscriber
RTP Stream
42
VXML
Gateway
SIP Service
Agent PG
ICM Service
EAPIM
Switch Leg
JTAPI GW
Cache
41
CVP Call Server
VXML over HTTP
VXML over HTTP
JTAPI
Client
CTI Manager
JTAPI
UCCE
VRU PG
39
39
VRU PIMs
Router
PM, HoldMusic, S
PM, HoldMusic, S
GED-125 (SCI)
DMP
40
42
IVRVXML
Service
Page
VRU Leg
VXML Server
Media
Server
38
ASR
Server
#CLUS
BRKCCT-2210
PM,HoldMusic,S
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
159
UCCE Call Flow
43. The VXML Gateway performs an HTTP submit to the IVR
Service with ‘success’.
44. This is forwarded to the ICM Service.
45. This is sent to ICM. The routing script continues along the
SUCCESS path of the Run External Script (PM,HoldMusic,S)
node. The script should continue looping, playing one or more
short audio files to the caller each time until an agent becomes
available.
PSTN
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
SIP
SIP
CUCM
Subscriber
RTP Stream
VXML
Gateway
SIP Service
Agent PG
ICM Service
EAPIM
Switch Leg
JTAPI GW
CVP Call Server
VXML over HTTP
VXML over HTTP
CTI Manager
JTAPI
VRU PG
45
http submit
result=success
JTAPI
Client
45
UCCE
Router
VRU PIMs
result=success
GED-125 (SCI)
43
DMP
44
IVR Service
VRU Leg
result=success
VXML Server
PM,HoldMusic,S
Media
Server
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
160
UCCE Call Flow
46. When an agent becomes available, ICM does the following:
a. Sends an ‘http disconnect’ command to the IVR Service
to tear down the VRU Leg of the call.
b. Sends a SIP ‘BYE’ to the SIP Service to instruct the
Ingress Gateway to tear down the RTP audio stream to
the VXML Gateway.
c. Instructs the SIP Service to send the call to the agent’s
extension.
PSTN
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
SIP
SIP
CUCM
Subscriber
RTP Stream
VXML
Gateway
SIP Service
Switch Leg
Agent PG
ICM Service
c
EAPIM
b
JTAPI GW
CVP Call Server
VXML over HTTP
VXML over HTTP
c
CTI Manager
JTAPI
VRU PG
b
http “Disconnect”
(tear down VRU Leg)
JTAPI
Client
VRU PIMs
a
GED-125 (SCI)
c
UCCE
b
Router
a
http“Connect”
“Disconnect”
SIP
“Bye”
(tear
down
VRU
Leg)
(tear
down
audio)
(to
agent
extension)
DMP
a
IVR Service
VRU Leg
VXML Server
Media
Server
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
161
UCCE Call Flow
47. The instructions are carried out:
a. The VRU leg of the call is torn down.
b. Sends a SIP ‘BYE’ through the SIP Proxy Server to the
Ingress Gateway, which tears down the RTP audio
stream to the VXML Gateway.
c. The SIP Service consults the SIP Proxy Server for the IP
address of the Communications Manager Subscriber to
which the agent’s phone extension is registered.
d. The SIP Service sends a SIP Invite to that UCM
Subscriber.
e. UCM forwards the SIP Invite to the agent’s IP Phone.
f. Once the agent’s IP phone returns a SIP OK, UCM
sends the IP address of that phone to the SIP Service.
PSTN
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
Subscriber
IP Address
SIP
b
RTP Stream
OK
f
SIP
b
RTP Stream
VXML
Gateway
g
c
SIP Service
d
e
Agent PG
ICM Service
EAPIM
Switch Leg
Redirect
Subscriber
SIP
“Bye”
SIP
Invite
Audio
Stream
IP
Address?
JTAPI GW
JTAPI
Client
JTAPI
Phone
SIP Invite
CTI Manager
IP Address
UCCE
VRU PG
Router
VRU PIMs
CVP Call Server
CUCM
Subscriber
VXML over HTTP
VXML over HTTP
GED-125 (SCI)
DMP
g.
IVR Service
a
VRU Leg
VXML Server
Media
Server
ASR
Server
#CLUS
The SIP Service sends a message to the Ingress
Gateway to send the caller’s RTP audio stream to this IP
address.
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
162
UCCE Call Flow
48. CTI Information is sent to the agent desktop via the Finesse
server. This information is used to “pop” applications, such as
CRM applications, and populate customer records and fields
within those applications.
PSTN
TDM/SIP
CUBE/TDM-IP
Gateway
SIP Proxy
CUSP
SIP
HTTP/HTTPS
Finesse
SIP
CSTA
(GED-188)
VXML
Gateway
SIP Service
CUCM
Subscriber
Agent PG
ICM Service
EAPIM
Switch Leg
JTAPI GW
JTAPI
Client
CTI Manager
JTAPI
UCCE
VRU PG
Router
VRU PIMs
CVP Call Server
VXML over HTTP
VXML over HTTP
GED-125 (SCI)
DMP
IVR Service
Media
Server
VXML Server
ASR
Server
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
163
Single Sign-on
(SSO)
Securing Unified Contact Center Enterprise
Cisco Identity Service (IdS) token management and encryption
•
Refresh Token Expiry: Long-lived token used to
obtain a new or renewed access token. CCE/IdS
will issue a new token upon expiry.
•
Authorization Code Expiry: Maximum time which
the users must present the authorization code to
the IdS server to get the OAuth tokens (access
token & refresh token) for resource access.
•
Access Token Expiry: Short-lived token that
provides access to a resource. CCE uses a
‘reference token’ whereas this token requires
validation between the IdS and protected
resource (Finesse, CUIC)
•
Encrypt Token: Encrypts tokens using AES-128
CBC HMAC SHA-256
o
o
#CLUS
BRKCCT-2210
AES + CBC provides strong encryption
HMAC + SHA-256 provides complex hashing
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
165
Securing Unified Contact Center Enterprise
Defining Single Sign-on for UCCE
• Refer to Appendix A for detailed SAML event flow from both the Agent and
Cisco IdS perspective.
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
166
Finesse User
SAML & Oauth flow for achieving SSO
in Finesse with IdS
IdP
IdS
1
(1) Agent/Supervisor accesses the Finesse desktop URL
2
(2) Finesse detects that authentication mode is SSO and redirects
the browser to IDS
3
(3) Browser sends the redirect authorize request to IDS.
4a
(4a) IdS detects user has invalid access token
4b
(4b) IdS redirects the browser to Idenity Provider (IdP)
4c
(4c) Browser sends SAML GET to IdP
5a
(5a) IdP provides login page for authenticating the user
(5b) User enters their credential
5b
(6a) IdP sends SAML assertion back to browser which has UID, IdP Cookie
6a
(6b) Browser sends SAML assertion to the IdS
(6c) IdS validates SAML assertion, creates the access token &
authcode and sends back to the Browser
6b
6c
(7) Browser issues GET of the Finesse desktop with access token
(8) Finesse gets the access token and validates it with IdS
7
8
10
(9) IdS sends back that token is valid
9
(10) Finesse checks user role and provides user access to resource
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
167
ECE Gadget in Finesse – Interaction diagram
IdP
ECE Gadget
Finesse
ECE Service
IdS
Browser
GetToken()
ECE Req with Token
GET /userInfo (token validation)
OK userID (token validation)
ECE response
#CLUS
BRKCCT-2210
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
168
Step-by-Step SAML Event Flow – User Endpoint
Initial Finesse Login Request
Step-by-Step SAML Event Flow – User Endpoint
Initial Finesse Login Request
No valid access token, redirect to IdP
Step-by-Step SAML Event Flow – User Endpoint
Initial Finesse Login Request
No valid access token, redirect to IdP
IdP login page
Step-by-Step SAML Event Flow – User Endpoint
Initial Finesse Login Request
No valid access token, redirect to IdP
IdP login page
SAML assertion
Step-by-Step SAML Event Flow – User Endpoint
Initial Finesse Login Request
Redirect back to Finesse with authorization code to IdS
No valid access token, redirect to IdP
IdP login page
SAML assertion
Step-by-Step SAML Event Flow – User Endpoint
Initial Finesse Login Request
Redirect back to Finesse with authorization code to IdS
No valid access token, redirect to IdP
IdP login page
SAML assertion
Agent enters their extension
Step-by-Step SAML Event Flow – Cisco IdS
Disable Cisco IdS token encryption
User requesting Finesse access and Finesse checking OAuth token with the IdS
Refresh and access tokens are not valid, IdS redirects user to IdP
Step-by-Step SAML Event Flow – Cisco IdS
This is the SAML request sent to the IdP
Here is the SAML response relayed to the Cisco IdS via the Agent’s browser
This is the SAML cookie!
You can view this cookie in SAML Tracer under the HTTP tab
Step-by-Step SAML Event Flow – Cisco IdS
Lastly, the Cisco IdS creates the OAuth tokens for resource access
Resource Access Granted!