General Troubleshooting Lab Guide - CCE
Shikhar Choudhary, CX
Alagusundaram Sockalingam, CX
Binu Santhakumaran, CX
1|Page
Learning Objectives
This lab session is for Cisco Contact Center Enterprise engineers interested in learning about
general UCCE troubleshooting and cradle to the grave call tracking and analysis.
Contact Center Enterprise is a vast solution and requires same amount of knowledge to
troubleshoot its various components including CVP, UCCE (Router/Logger/PG/AW), Finesse etc.
This guide will help you troubleshoot various components of CCE.
Upon completion of this lab, you will be able to troubleshoot:





UCCE pre-routed call flow with CVP.
Certificates/Smart licensing
CUIC-Live data
Finesse failovers
UCCE Failovers
Pre-requisite knowledge
This is an intermediate session intended for engineers with prior Contact Center Enterprise
experience.
Disclaimer
This lab is not designed for a production system, thus not all recommended features are
implemented or enabled optimally. For implementation and design-related questions, please
contact your representative at Cisco, or a Cisco partner.
2|Page
Table of Contents
Learning Objectives ..................................................................................................................... 2
Pre-requisite knowledge ............................................................................................................. 2
Disclaimer ................................................................................................................................... 2
Lab Topology and Access ........................................................................................................... 5
Lab Deployment .......................................................................................................................... 5
Lab Addressing and Credentials ...................................................................................................... 5
Agent and Customer Information .................................................................................................... 6
UCCE Overview ........................................................................................................................... 7
Cisco Unified Contact Center Enterprise Components............................................................... 7
CVP Pre-routed Call Flow – Internal Voice ................................................................................ 7
Lab 1: Login to Finesse ............................................................................................................... 8
Exercise 1: Login to Finesse Agent Desktop TEST .................................................................... 8
Lab 2: Configure PCCE to Execute HelloWorld ........................................................................... 11
Exercise 1: Connect to WKST1 ............................................................................................... 12
Lab 3: Log Analysis for CVP Pre-routed Call Flow ...................................................................... 22
Exercise 1: Connect to WKST1 RDP ....................................................................................... 22
Lab 4: UCCE Certificates (CA Signed)........................................................................................ 44
Exercise 1: Verify Certificates on UCCE Servers ...................................................................... 44
Lab 5: Troubleshoot Smart Licensing. ........................................................................................ 48
Exercise 1: Login to WKST1 and RDP to AW ........................................................................... 48
Lab 6: Troubleshoot CUIC and Live Data .................................................................................... 51
Exercise 1: Login to WKST1 and RDP to ROGGER, PG and CUIC ............................................. 51
Lab 7: Troubleshoot Finesse ...................................................................................................... 54
Exercise 1: Verify current logging levels.................................................................................. 54
Exercise 2: Collect Finesse Logs via CLI or RTMT or Web ....................................................... 57
Exercise 3: Finesse PG Failover .............................................................................................. 64
Exercise 4: Fun with Finesse API’s .......................................................................................... 65
Exercise 4: Few Finesse CLI Commands................................................................................. 70
Lab 8: Troubleshoot Router/PG and Collect Logs ....................................................................... 71
Exercise 1: Login and execute RTTEST and dumplog .............................................................. 71
Exercise 2: How to collect router process log using Diagnostic framework portico? ................ 73
Exercise 3: How to collect pg process log using Diagnostic framework portico? ..................... 75
3|Page
Lab 9: Troubleshoot VVB ........................................................................................................... 76
Exercise 1: CLI Commands..................................................................................................... 76
Exercise 2: VVB Logs ............................................................................................................. 81
4|Page
Lab Topology and Access
Lab Deployment
Lab Addressing and Credentials
Server
IP Address
OS Login - Password
WKST1
198.18.133.75
DCLOUD\sjeffers – C1sco12345
WKST2
198.18.133.76
DCLOUD\rbarrows – C1sco12345
ROGGER
198.18.135.37
DCLOUD\Administrator – C1sco12345
AW
198.18.133.11
DCLOUD\Administrator – C1sco12345
PG
198.18.133.12
DCLOUD\Administrator – C1sco12345
CVP
198.18.133.13
DCLOUD\Administrator – C1sco12345
ECEWEB
198.18.135.3
DCLOUD\Administrator – C1sco12345
CUIC
198.18.133.15
administrator – dCloud!23
Finesse
198.18.133.16
administrator – dCloud!23
CUCM
198.18.133.3
administrator – C1sco12345
VVB
198.18.133.143
administrator – dCloud!23
CUBE
198.18.133.226
admin – C1sco12345
RP
198.18.135.68
administrator – C1sco12345
Note: The lab pod is not configured and sized according to SRND production requirements. It was
created using VM clones of a master pod with the goal of hosting multiple identical pods. These
5|Page
labs are for instructional purposes only; please do not consider the deployment model and
allocated hardware resources as a reference for a production system.
Agent and Customer Information
6|Page
Agent Name
Username
Password
Agent ID
Agent Extension
Sandra Jefferson
sjeffers
C1sco12345
sjeffers
1080
Customer Name
Username
Password
Agent ID
Agent Extension
Rick Barrows
rbarrows
C1sco12345
rbarrows
1082
UCCE Overview
Cisco Unified Contact Center Enterprise (Unified CCE) and Package Unified Contact Center
Enterprise are a solution that delivers intelligent call routing, network-to-desktop Computer
Telephony Integration (CTI), and multichannel contact management to contact center agents over
an IP network.
Cisco Unified Contact Center Enterprise Components
The Unified CCE product integrates with Cisco Unified Communications Manager, Cisco Unified
Customer Voice Portal, Cisco VoIP Gateways, and Cisco Unified IP Phones. Together these
products provide a contact center solution to achieve intelligent call routing, multichannel ACD
functionality, voice response unit (VRU) functionality, network call queuing, and consolidated
enterprise-wide reporting. Many additional options and components are available. On these lab
exercises you may use the following components:







Ingress, Egress, Cisco Unfired Border Element (CUBE) Gateways
Unified Customer Voice Portal (CVP)
Package Contact Center Enterprise (PCCE)
Cisco Virtualized Voice Browser (VVB)
Cisco Unified Communications Manager (CUCM)
Cisco Finesse
Cisco Unified Intelligence Center (CUIC)
CVP Pre-routed Call Flow – Internal Voice
The CVP pre-routed call flow model a very similar experience to a CVP comprehensive call flow
and is perfect for lab testing. This is the most common call flow model implemented in UCCE for
internal call transfers or simply internally initiated calls. There are several protocols involved as
part of the exchange messages between the components. You can see this message exchange
and call flow in the image and steps below.
7|Page
Call is Initiated from any IP Phone to a CTI Route Point registered to UCCE PG:
1. Call from IP Phone to a CTI Route Point registered to UCCE.
2. UCCE Router provides a label to redirect call to CVP and CUCM redirects the call to CVP.
3. CVP receives the call and receives a VRU label.
4. CVP sends call to CVVB, caller hears IVR.
HTTP
5. Agent is available now, PCCE sends the Agent extension to CVP via VRU PG.
6. CVP sends call to an agent via CUCM.
Lab 1: Login to Finesse
In this lab you will test Finesse login before making necessary changes to route calls to
agents.
Relevant Topology Information
Server
WKST1
IP Address
198.18.133.75
OS Login - Password
DCLOUD\sjeffers – C1sco12345
Exercise 1: Login to Finesse Agent Desktop TEST
Step 1. On WKST1, open Firefox and navigate to the second tab .
8|Page
Step 2. Click the Demo Links menu and select Finesse Agent Desktop.
Step 3. Keep English as the language and click Next.
9|Page
Step 4. As Username type sjeffers and click Next.
Step 5. In the SSO window, leave the default values and click Sign in.
Step 6. For the Extension type 1080 and click Submit.
10 | P a g e
Step 7. We have now the agent logged in to Finesse.
Lab 2: Configure PCCE to Execute HelloWorld
In this section, you will configure PCCE to execute HelloWorld VXML Application when callers call
DN 7704 from CUCM. You will also create a basic skill/call type and associate scripts to dialed
numbers.
Relevant Topology Information
Server
IP Address
OS Login - Password
WKST1
198.18.133.75
DCLOUD\sjeffers – C1sco12345
AW
198.18.133.11
DCLOUD\Administrator – C1sco12345
WKST2
198.18.133.76
DCLOUD\rbarrows – C1sco12345
11 | P a g e
Exercise 1: Connect to WKST1
Step 1. From WKST1, open Firefox and login to CCEAdmin:
Step 2. Click on CallType under the Call Settings tile.
12 | P a g e
Step 3. Create a Calltype.
Step 4. Create a Skill under Organization Setup -> Skill Group section.
Step 5. RDP to AW and open Script Editor and open HelloWorld script.
13 | P a g e
Step 6. Add two nodes. Line Connector and Queue.
Step 7. Modify Queue to Skill Group node as seen in the image.
14 | P a g e
Step 8. Final script should look like this. Call will queue to your skill group and continue to be at
VRU executing HelloWorld in a loop till an agent is available.
Step 9. Click on Script menu and Validate the script of any errors. Save the Script.
15 | P a g e
Step 10. Under Script Menu, select Display Node ID’s and then Monitor Script.
Step 11. Navigate to Script -> CallType Manager.
16 | P a g e
Step 12. Select the CallType from the dropdown and click on Add button.
Scroll to HellowWorld script and click on OK.
Step 13. On CCEAdmin, click on Call Settings Tile and Route Settings.
Step 14. Navigate to Dialed Number and search for 7704.
17 | P a g e
Step 15. Edit them one by one and set CallType to LabCall:
Step 16. RDP to WKST2 and from Jabber, make a call to 7704.
18 | P a g e
Call should connect and you should hear:
“You have successfully installed the Cisco Unified VXML Server”.
Step 17. On CCEAdmin, Click on User Setup -> Agents and search sjeffers.
Step 18. Navigate to Skill Groups tab and add LabFlowSkill and click on Save.
19 | P a g e
Step 19. Log in sjeffers on Finesse agent Desktop via WKST1 if logged out. Use extension 1080
You can see LabFlowSkill on QueueStatistics gadget and agent is in Not Ready state:
20 | P a g e
Step 20. Using WKST2, call 7704 again and remain in queue with HelloWorld within the loop.
Using WKST1, change state of sjeffers to Ready and call should land on Finesse desktop with
Agent status as Reserved.
Step 21. Click on Answer and Agent will move to Talking State. Use End call button to end the call
and agent will move to a Wrap-up state.
21 | P a g e
Lab 3: Log Analysis for CVP Pre-routed Call Flow
In this lab you will learn how to follow the call through a CVP pre-routed call.
Relevant Topology Information
Server
WKST1
IP Address
198.18.133.75
OS Login - Password
DCLOUD\sjeffers – C1sco12345
Exercise 1: Connect to WKST1 RDP
Step 1. From WKST1, open mRemoteNG and you can RDP to relevant servers from there on.
22 | P a g e
Enable CVP Debug logs:
Under Serv Mgr dropdown select com.dynamicsoft.DsLibs.DsUALibs and under Level select Debug. Click
on Set Button and then Click on Debug/41
Make a call to HelloWorld. Verify Call landed on CVP from CUCM. Remote shows the IP and port
of the Ingress device by collecting CVP Call server logs. Search for your ANI.
8740982: 198.18.133.13: Sep 20 2022 11:24:48.513 -0400: %_TransactionManagement-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.TransactionManagement: processMessage():
Incoming message binding info: TCP local[[ port = 5060 (198.18.133.13)]] remote[[ port = 45743
(198.18.133.3) ]], Connection ID = null, network = DEFAULT, TSIP = false
8740983: 198.18.133.13: Sep 20 2022 11:24:48.513 -0400: %_TransactionManagement-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.TransactionManagement: processMessage():
Incoming message:
INVITE sip:88811110001365@198.18.133.13:5060 SIP/2.0
Via: SIP/2.0/TCP 198.18.133.3:5060;branch=z9hG4bK2de21136dc9ee
Max-Forwards: 69
To: <sip:88811110001365@198.18.133.13>
From: <sip:1082@198.18.133.3>;tag=115683~67f921d2-679e-4fdb-a0e9-0df0d4a1fe0a25046557
23 | P a g e
Call-ID: 5bae6580-3291db40-8a6f-38512c6@198.18.133.3
CSeq: 101 INVITE
Content-Length: 1405
Date: Tue, 20 Sep 2022 15:24:48 GMT
Supported: timer,resource-priority,replaces
Supported: X-cisco-srtp-fallback
Supported: Geolocation
Min-SE: 1800
User-Agent: Cisco-CUCM11.5
Allow: INVITE, OPTIONS, INFO, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE,
NOTIFY
Expires: 180
Allow-Events: presence
Call-Info: <urn:x-cisco-remotecc:callinfo>;x-cisco-video-traffic-class=DESKTOP
Session-ID:
00007c6900105000a000005056a06cb9;remote=00000000000000000000000000000000
Cisco-Guid: 1538155904-0000065536-0000024011-0059052742
Session-Expires: 1800
P-Asserted-Identity: <sip:1082@198.18.133.3>
Remote-Party-ID: <sip:1082@198.18.133.3>;party=calling;screen=yes;privacy=off
Contact:
<sip:1082@198.18.133.3:5060;transport=tcp>;video;audio;+u.sip!devicename.ccm.cisco.com="U
CSFRBARROWS";bfcp
Content-Type: application/sdp
v=0
o=CiscoSystemsCCM-SIP 115683 1 IN IP4 198.18.133.3
s=SIP Call
c=IN IP4 198.18.133.76
b=TIAS:768000
b=AS:768
t=0 0
a=cisco-mari:v1
a=cisco-mari-rate
m=audio 22894 RTP/AVP 0 8 18 104 105 114 111 101
a=extmap:14/sendrecv http://protocols.cisco.com/timestamp#100us
24 | P a g e
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:104 G7221/16000
a=fmtp:104 bitrate=32000
a=rtpmap:105 G7221/16000
a=fmtp:105 bitrate=24000
a=rtpmap:114 opus/48000/2
a=rtpmap:111 X-ULPFECUC/8000
a=fmtp:111 max_esel=1420;m=8;max_n=32;FEC_ORDER=FEC_SRTP
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
m=video 25680 RTP/AVP 126 97 111
b=TIAS:768000
a=label:11
a=extmap:14/sendrecv http://protocols.cisco.com/timestamp#100us
a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42801E;packetization-mode=1;max-fs=1621;max-rcmd-nalusize=32000;level-asymmetry-allowed=1
a=imageattr:126 recv [x=[32:1:1280],y=[18:1:720],par=1.7778,q=1.00]
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42801E;packetization-mode=0;max-fs=1621;level-asymmetryallowed=1
a=imageattr:97 recv [x=[32:1:1280],y=[18:1:720],par=1.7778,q=1.00]
a=rtpmap:111 X-ULPFECUC/90000
a=fmtp:111 max_esel=1420;m=8;max_n=32;FEC_ORDER=FEC_SRTP
a=content:main
a=rtcp-fb:* ccm fir
a=rtcp-fb:* ccm tmmbr
a=rtcp-fb:* nack pli
m=application 31158 RTP/AVP 125
a=rtpmap:125 H224/4800
a=rtcp:31159
25 | P a g e
Verify and note down the GUID’s CVP creates from initial GUID:
1781461: 198.18.133.13: Sep 20 2022 11:24:48.513 -0400: %CVP_12_6_SIP-7-CALL:
{Thrd=DATAI.4} NEW CALL with guid=5BAE65800001000000005DCB038512C6
legid=5bae6580-3291db40-8a6f-38512c6 dn=88811110001365 ani=1082 uui=null calldate=Tue
Sep 20 11:24:48 EDT 2022 video=true cachecallcontext = false is_postcallsurvey = false
RouterCallKey = null RouterCallKeyDay = null RouterCallKeySequenceNumber = null
Verify VRU Label from UCCE and ensure it is correct and that you have routes in CVP:
316317: 198.18.133.13: Sep 20 2022 11:24:48.513 -0400: %CVP_12_6_ICM-7-CALL:
{Thrd=Event-router-group-0} CALLGUID=5BAE65800001000000005DCB038512C6
DialogId=29454 [SIP_CORRID_LEG_DIALOGUE] - Processing from SIP_SS - [NEW_CALL]
DialogID=0 SendSeqNo=0 DNIS=88811110001365 ANI=1082 CED= rckey=0 rcday=0 rcseq=0
uui= callguid=5BAE65800001000000005DCB038512C6 trunkGroupId=0 trunkNumber=0
serviceId=0 calledNumber= location= locationpkid= pstntrunkgroupid= sipheader= ECC variables=
ECC arrays= Peripheral Variables=
316318: 198.18.133.13: Sep 20 2022 11:24:48.513 -0400: %CVP_12_6_ICM-7-CALL:
{Thrd=Event-router-group-0} CALLGUID=5BAE65800001000000005DCB038512C6
DialogId=29454 [SIP_CORRID_LEG_DIALOGUE] - Publishing to UCCE [REQUEST_INSTRUCTION] DialogID=29454 SendSeqNo=1
CALLGUID=5BAE65800001000000005DCB038512C6 ANI=1082 DNIS=88811110001365
correlationId=1365 trunkGroupId=200 trunkNumber=0 serviceId=2 uui= calledNumber= location=
locationpkid= pstntrunkgroupid=198.18.133.3 pstntrunkgroupchannelnum=2147483647
sipheader=
316319: 198.18.133.13: Sep 20 2022 11:24:48.513 -0400: %CVP_12_6_ICM-7-CALL:
{Thrd=Event-router-group-0} CALLGUID=5BAE65800001000000005DCB038512C6
DialogId=29454 [SIP_CORRID_LEG_DIALOGUE] - Processing from UCCE - ,,
[ICM_TEMPORARY_CONNECT], rckey=2218, rcday=154029, rcseq=2, DialogID=29454,
SendSeqNo=1, callguid=5BAE65800001000000005DCB038512C6, label=7771111000,
correlationId=1366, location=, locationpkid=, sipheader=, ECC variables=user.microapp.locale:
en-us,user.microapp.app_media_lib: ..,user.microapp.media_server:
http://198.18.133.13:7000/CVP,user.microapp.UseVXMLParams: N, ECC
arrays=USER_MICROAPP_TOEXTVXML[0]:application=HelloWorld,, Peripheral Variables=,
DNIS=88811110001365, ANI=1082
316320: 198.18.133.13: Sep 20 2022 11:24:48.513 -0400: %CVP_12_6_ICM-7-CALL:
{Thrd=Event-router-group-0} CALLGUID=5BAE65800001000000005DCB038512C6
DialogId=29454 [SIP_CORRID_LEG_DIALOGUE] - Publishing to SIP_SS - [CONNECT]
DialogID=29454 SendSeqNo=0, Label=7771111000, TxrHint=false, rckey=2218,
rcday=154029, rcseq=2, location=, locationpkid=, sipheader=, uui=, whisperAnnounce=null,
callguid=5BAE65800001000000005DCB038512C6, serviceparam=, ECC
variables=user.microapp.locale: en-us,user.microapp.app_media_lib:
..,user.microapp.media_server: http://198.18.133.13:7000/CVP,user.microapp.UseVXMLParams:
N, ECC arrays=USER_MICROAPP_TOEXTVXML[0]:application=HelloWorld,, Peripheral Variables=,
DNIS=88811110001365, ANI=1082
26 | P a g e
Verify CVP was able to Route the Call to VVB and note down the GUID:
8741039: 198.18.133.13: Sep 20 2022 11:24:48.535 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message binding info (NB):
TCP local[[ port = 0 (198.18.133.13)]] remote[[ port = 5060 (198.18.133.143) ]], Connection ID =
null, network = DEFAULT, TSIP = false
8741040: 198.18.133.13: Sep 20 2022 11:24:48.535 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message (NB): INVITE
sip:77711110001366@vvb1.dcloud.cisco.com;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 198.18.133.13:5060;branch=z9hG4bK6VD7zVvGnoOa4c28s+rKVQ~~148148
Max-Forwards: 68
To: <sip:77711110001366@vvb1.dcloud.cisco.com;transport=tcp>
From: 1082 <sip:1082@198.18.133.13:5060>;tag=dsf1ad117
Call-ID: 5BAE65800001000000005DCB038512C6-166368748853232166@198.18.133.13
CSeq: 1 INVITE
Content-Length: 1413
Contact: <sip:1082@198.18.133.13:5060;transport=tcp>;audio;video
Expires: 60
User-Agent: CVP 12.6 (1) ES-10 Build-50
Call-Info: <sip:198.18.133.3:5060>;purpose=x-cisco-origIP
Call-Info: <urn:x-cisco-remotecc:callinfo>;x-cisco-video-traffic-class=DESKTOP
Date: Tue, 20 Sep 2022 15:24:48 GMT
Min-SE: 1800
Allow: INVITE, OPTIONS, INFO, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE,
NOTIFY
Allow-Events: presence
Session-ID:
00007c6900105000a000005056a06cb9;remote=00000000000000000000000000000000
Cisco-Guid: 1538155904-0000065536-0000024011-0059052742
Session-Expires: 1800
P-Asserted-Identity: <sip:1082@198.18.133.3>
Remote-Party-ID: <sip:1082@198.18.133.3>;party=calling;screen=yes;privacy=off
Cisco-Gucid: 5BAE65800001000000005DCB038512C6
Supported: timer
Supported: resource-priority
Supported: replaces
Supported: X-cisco-srtp-fallback
27 | P a g e
Supported: Geolocation
Supported: sdp-anat
Content-Type: application/sdp
App-Info: <198.18.133.13:7000:7443>
v=0
o=Cisco-CVP-B2BUA 289844526 289844526 IN IP4 198.18.133.13
s=SIP Call
c=IN IP4 198.18.133.76
b=TIAS:768000
b=AS:768
t=0 0
a=cisco-mari:v1
a=cisco-mari-rate
m=audio 22894 RTP/AVP 0 8 18 104 105 114 111 101
a=extmap:14/sendrecv http://protocols.cisco.com/timestamp#100us
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:104 G7221/16000
a=fmtp:104 bitrate=32000
a=rtpmap:105 G7221/16000
a=fmtp:105 bitrate=24000
a=rtpmap:114 opus/48000/2
a=rtpmap:111 X-ULPFECUC/8000
a=fmtp:111 max_esel=1420;m=8;max_n=32;FEC_ORDER=FEC_SRTP
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
m=video 25680 RTP/AVP 126 97 111
b=TIAS:768000
a=label:11
a=extmap:14/sendrecv http://protocols.cisco.com/timestamp#100us
a=rtpmap:126 H264/90000
28 | P a g e
a=fmtp:126 profile-level-id=42801E;packetization-mode=1;max-fs=1621;max-rcmd-nalusize=32000;level-asymmetry-allowed=1
a=imageattr:126 recv [x=[32:1:1280],y=[18:1:720],par=1.7778,q=1.00]
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42801E;packetization-mode=0;max-fs=1621;level-asymmetryallowed=1
a=imageattr:97 recv [x=[32:1:1280],y=[18:1:720],par=1.7778,q=1.00]
a=rtpmap:111 X-ULPFECUC/90000
a=fmtp:111 max_esel=1420;m=8;max_n=32;FEC_ORDER=FEC_SRTP
a=content:main
a=rtcp-fb:* ccm fir
a=rtcp-fb:* ccm tmmbr
a=rtcp-fb:* nack pli
m=application 31158 RTP/AVP 125
a=rtpmap:125 H224/4800
a=rtcp:31159
Verify and note that VVB sends a 200 OK post 100 Trying Etc with IP Address and RTP port:
c=IN IP4 198.18.133.143 and m=audio 26812 RTP/AVP 0 101:
8741060: 198.18.133.13: Sep 20 2022 11:24:48.550 -0400: %_TransactionManagement-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.TransactionManagement: processMessage():
Incoming message binding info: TCP local[[ port = 0 (198.18.133.13)]] remote[[ port = 5060
(198.18.133.143) ]], Connection ID = null, network = DEFAULT, TSIP = false
8741061: 198.18.133.13: Sep 20 2022 11:24:48.550 -0400: %_TransactionManagement-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.TransactionManagement: processMessage():
Incoming message:
SIP/2.0 200 Ok
Via: SIP/2.0/TCP 198.18.133.13:5060;branch=z9hG4bK6VD7zVvGnoOa4c28s+rKVQ~~148148
To: <sip:77711110001366@vvb1.dcloud.cisco.com;transport=tcp>;tag=dsdca4d2ed
From: 1082 <sip:1082@198.18.133.13:5060>;tag=dsf1ad117
Call-ID: 5BAE65800001000000005DCB038512C6-166368748853232166@198.18.133.13
CSeq: 1 INVITE
Content-Length: 194
Contact: <sip:77711110001366@198.18.133.143:5060;transport=tcp>
Content-Type: application/sdp
Supported: timer
29 | P a g e
Session-Expires: 1800;refresher=uac
Require: timer
Allow: INVITE, INFO, BYE, CANCEL, ACK, UPDATE
Server: Cisco-VVB-12.5
Session-ID:
5B8075D839ED7A710001018110000183;remote=00007c6900105000a000005056a06cb9
v=0
o=Cisco-CRS 1514 1 IN IP4 198.18.133.143
s=SIP Call
c=IN IP4 198.18.133.143
t=0 0
m=audio 26812 RTP/AVP 0 101
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
Verify CVP sent the same SDP to Ingress (CUCM):
8741079: 198.18.133.13: Sep 20 2022 11:24:48.566 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message binding info (NB):
TCP local[[ port = 5060 (198.18.133.13)]] remote[[ port = 45743 (198.18.133.3) ]], Connection ID
= null, network = DEFAULT, TSIP = false
8741080: 198.18.133.13: Sep 20 2022 11:24:48.566 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message (NB): SIP/2.0 200
Ok
Via: SIP/2.0/TCP 198.18.133.3:5060;branch=z9hG4bK2de21136dc9ee
To: <sip:88811110001365@198.18.133.13>;tag=dscc4bd129
From: <sip:1082@198.18.133.3>;tag=115683~67f921d2-679e-4fdb-a0e9-0df0d4a1fe0a25046557
Call-ID: 5bae6580-3291db40-8a6f-38512c6@198.18.133.3
CSeq: 101 INVITE
Content-Length: 1046
Cisco-Guid: 1538155904-0000065536-0000024011-0059052742
Contact: <sip:198.18.133.13:5060;transport=tcp>
Session-Expires: 1800;refresher=uac
Allow: INVITE, INFO, BYE, CANCEL, ACK, UPDATE
30 | P a g e
Server: Cisco-VVB-12.5
Session-ID:
5B8075D839ED7A710001018110000183;remote=00007c6900105000a000005056a06cb9
Require: timer
Supported: timer
Supported: sdp-anat
Content-Type: application/sdp
Cisco-Gucid: 5BAE65800001000000005DCB038512C6
v=0
o=Cisco-CVP-B2BUA 289844526 289844526 IN IP4 198.18.133.13
s=SIP Call
c=IN IP4 198.18.133.143
t=0 0
m=audio 26812 RTP/AVP 0 101
c=IN IP4 198.18.133.143
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
m=video 0 RTP/AVP 126 97 111
c=IN IP4 198.18.133.76
b=TIAS:768000
a=label:11
a=extmap:14/sendrecv http://protocols.cisco.com/timestamp#100us
a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42801E;packetization-mode=1;max-fs=1621;max-rcmd-nalusize=32000;level-asymmetry-allowed=1
a=imageattr:126 recv [x=[32:1:1280],y=[18:1:720],par=1.7778,q=1.00]
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42801E;packetization-mode=0;max-fs=1621;level-asymmetryallowed=1
a=imageattr:97 recv [x=[32:1:1280],y=[18:1:720],par=1.7778,q=1.00]
a=rtpmap:111 X-ULPFECUC/90000
a=fmtp:111 max_esel=1420;m=8;max_n=32;FEC_ORDER=FEC_SRTP
a=content:main
31 | P a g e
a=rtcp-fb:* ccm fir
a=rtcp-fb:* ccm tmmbr
a=rtcp-fb:* nack pli
m=application 0 RTP/AVP 125
c=IN IP4 198.18.133.76
a=rtpmap:125 H224/4800
a=rtcp:31159
Verify CVP Receives a Request in VXML logs via VVB HTTP:
246623: 198.18.133.13: Sep 20 2022 11:24:48.582 -0400: %CVP_12_6_VXML_SERVER-6VXML_SERVER_CALLGUID_INFO: [NEW_CALL]- Received callid =
5BAE65800001000000005DCB038512C6 , legid = 5BAE65800001000000005DCB038512C6166368748853232166@198.18.133.13 [id:6021]
Verify CVP VXML logs. You can see Session ID and License in Use (c:\cisco\cvp\logs\VXML\):
246631: 198.18.133.13: Sep 20 2022 11:24:48.597 -0400: %CVP_12_6_VXML_SERVER-7CALL: {Thrd=http-processor50} LicenseManagerSessionObject:valueBound:
CALLGUID=5BAE65800001000000005DCB038512C6 - License obtained; JSessionID=*****
SessionID=198.18.133.13.1663687488597.11187.HelloWorld Licenses in Use=1
Note when VXML Server finished processing and license was released:
246635: 198.18.133.13: Sep 20 2022 11:24:56.723 -0400: %CVP_12_6_VXML_SERVER-7LOW_LEVEL: {Thrd=Global-task-executor-group-0} [Removed - 0]
246636: 198.18.133.13: Sep 20 2022 11:24:57.391 -0400: %CVP_12_6_VXML_SERVER-7CALL: {Thrd=http-processor49} LicenseManagerSessionObject:valueUnbound: - License
released; JSessionID=***** Licenses in Use=0
246637: 198.18.133.13: Sep 20 2022 11:24:57.406 -0400: %CVP_12_6_VXML_SERVER-6VXML_SERVER_CALLGUID_INFO: [CALL_RESULT]- Received
callid=5BAE65800001000000005DCB038512C6 [id:6021]
32 | P a g e
Verify the label provided by UCCE to route to agent:
316340: 198.18.133.13: Sep 20 2022 11:25:02.601 -0400: %CVP_12_6_ICM-7-CALL:
{Thrd=Event-router-group-0} CALLGUID=5BAE65800001000000005DCB038512C6
DialogId=29454 [SIP_CORRID_LEG_DIALOGUE] - Processing from UCCE - [CONNECT]
DialogID=29454 SendSeqNo=2, Label=1080, TxrHint=false, rckey=2218, rcday=154029,
rcseq=2, location=, locationpkid=, sipheader=, uui=, whisperAnnounce=,
callguid=5BAE65800001000000005DCB038512C6,
serviceparam={"Services":{"ConfigId":"4f62417b-bbea-44b9-9ba0ba3e51ad8272","Enabled":["AgentAnswers","Transcription"]}}, ECC
variables=user.microapp.error_code: 0,user.microapp.locale: enus,user.microapp.app_media_lib: ..,user.microapp.media_server:
http://198.18.133.13:7000/CVP,user.microapp.caller_input: ****,user.microapp.UseVXMLParams:
N, ECC arrays=USER_MICROAPP_TOEXTVXML[0]:application=HelloWorld,, Peripheral Variables=,
DNIS=88811110001365, ANI=1082
From here on, CVP is ending VXML session and negotiating media between caller and agent.
CVP Sends a BYE to VXML Leg:
8741538: 198.18.133.13: Sep 20 2022 11:25:02.623 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message binding info (NB):
TCP local[[ port = 0 (198.18.133.13)]] remote[[ port = 5060 (198.18.133.143) ]], Connection ID =
null, network = DEFAULT, TSIP = false
8741539: 198.18.133.13: Sep 20 2022 11:25:02.623 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message (NB): BYE
sip:77711110001366@198.18.133.143:5060;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 198.18.133.13:5060;branch=z9hG4bK6VD7zVvGnoOa4c28s+rKVQ~~148155
Max-Forwards: 70
To: <sip:77711110001366@vvb1.dcloud.cisco.com;transport=tcp>;tag=dsdca4d2ed
From: <sip:1082@198.18.133.13:5060>;tag=dsf1ad117
Call-ID: 5BAE65800001000000005DCB038512C6-166368748853232166@198.18.133.13
CSeq: 2 BYE
Content-Length: 0
Contact: <sip:1082@198.18.133.13:5060;transport=tcp>;audio;video
Reason: Q.850;cause=16
Session-ID:
00000000000000000000000000000000;remote=5B8075D839ED7A710001018110000183
33 | P a g e
CVP Sends an Invite to Ringback leg:
8741572: 198.18.133.13: Sep 20 2022 11:25:02.623 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message binding info (NB): TCP local[[
port = 0 (198.18.133.13)]] remote[[ port = 5060 (198.18.133.143) ]], Connection ID = null, network =
DEFAULT, TSIP = false
8741573: 198.18.133.13: Sep 20 2022 11:25:02.623 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message (NB): INVITE
sip:91919191@vvb1.dcloud.cisco.com;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 198.18.133.13:5060;branch=z9hG4bK6VD7zVvGnoOa4c28s+rKVQ~~148156
Max-Forwards: 68
To: <sip:91919191@vvb1.dcloud.cisco.com;transport=tcp>
From: 1082 <sip:1082@198.18.133.13:5060>;tag=dsa9ed822
Call-ID: 5BAE65800001000000005DCB038512C6-166368750262332167@198.18.133.13
CSeq: 1 INVITE
Content-Length: 0
Contact: <sip:1082@198.18.133.13:5060;transport=tcp>;audio;video
Expires: 10
User-Agent: CVP 12.6 (1) ES-10 Build-50
Call-Info: <sip:198.18.133.3:5060>;purpose=x-cisco-origIP
Call-Info: <urn:x-cisco-remotecc:callinfo>;x-cisco-video-traffic-class=DESKTOP
Date: Tue, 20 Sep 2022 15:24:48 GMT
Min-SE: 1800
Allow: INVITE, OPTIONS, INFO, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY
Allow-Events: presence
Session-ID: 00007c6900105000a000005056a06cb9;remote=00000000000000000000000000000000
Session-Expires: 1800
P-Asserted-Identity: <sip:1082@198.18.133.3>
Remote-Party-ID: <sip:1082@198.18.133.3>;party=calling;screen=yes;privacy=off
Cisco-Gucid: 5BAE65800001000000005DCB038512C6
Supported: timer
Supported: resource-priority
Supported: replaces
Supported: X-cisco-srtp-fallback
Supported: Geolocation
Supported: sdp-anat
App-Info: <198.18.133.13:7000:7443>;ringtone=ringback.wav
34 | P a g e
CVP receives 200 OK for Ringback leg:
8741611: 198.18.133.13: Sep 20 2022 11:25:02.639 -0400: %_TransactionManagement-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.TransactionManagement: processMessage(): Incoming
message binding info: TCP local[[ port = 0 (198.18.133.13)]] remote[[ port = 5060 (198.18.133.143) ]],
Connection ID = null, network = DEFAULT, TSIP = false
8741612: 198.18.133.13: Sep 20 2022 11:25:02.639 -0400: %_TransactionManagement-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.TransactionManagement: processMessage(): Incoming
message:
SIP/2.0 200 Ok
Via: SIP/2.0/TCP 198.18.133.13:5060;branch=z9hG4bK6VD7zVvGnoOa4c28s+rKVQ~~148156
To: <sip:91919191@vvb1.dcloud.cisco.com;transport=tcp>;tag=ds8f219f8
From: 1082 <sip:1082@198.18.133.13:5060>;tag=dsa9ed822
Call-ID: 5BAE65800001000000005DCB038512C6-166368750262332167@198.18.133.13
CSeq: 1 INVITE
Content-Length: 177
Contact: <sip:91919191@198.18.133.143:5060;transport=tcp>
Content-Type: application/sdp
Supported: timer
Session-Expires: 1800;refresher=uac
Require: timer
Allow: INVITE, INFO, BYE, CANCEL, ACK, UPDATE
Server: Cisco-VVB-12.5
Session-ID: 5B80ACE239ED7A710001018310000183;remote=00007c6900105000a000005056a06cb9
v=0
o=Cisco-CRS 3316 1 IN IP4 198.18.133.143
s=SIP Call
c=IN IP4 198.18.133.143
t=0 0
m=audio 26814 RTP/AVP 0 101
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
35 | P a g e
CVP sends that media information to Ingress (CUCM):
8741656: 198.18.133.13: Sep 20 2022 11:25:02.655 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message binding info (NB): TCP local[[
port = 0 (198.18.133.13)]] remote[[ port = 5060 (198.18.133.3) ]], Connection ID = null, network =
DEFAULT, TSIP = false
8741657: 198.18.133.13: Sep 20 2022 11:25:02.655 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message (NB): INVITE
sip:1082@198.18.133.3:5060;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 198.18.133.13:5060;branch=z9hG4bK6VD7zVvGnoOa4c28s+rKVQ~~148157
Max-Forwards: 70
To: <sip:1082@198.18.133.3>;tag=115683~67f921d2-679e-4fdb-a0e9-0df0d4a1fe0a-25046557
From: <sip:88811110001365@198.18.133.13>;tag=dscc4bd129
Call-ID: 5bae6580-3291db40-8a6f-38512c6@198.18.133.3
CSeq: 1 INVITE
Content-Length: 1029
Contact: <sip:198.18.133.13:5060;transport=tcp>
Session-ID: 5B80ACE239ED7A710001018310000183;remote=00000000000000000000000000000000
Require: timer
Supported: timer
Supported: sdp-anat
Content-Type: application/sdp
Session-Expires: 1800;refresher=uas
v=0
o=Cisco-CVP-B2BUA 289844526 289844527 IN IP4 198.18.133.13
s=SIP Call
c=IN IP4 198.18.133.143
t=0 0
m=audio 26814 RTP/AVP 0 101
c=IN IP4 198.18.133.143
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
m=video 0 RTP/AVP 126 97 111
c=IN IP4 198.18.133.76
b=TIAS:768000
a=label:11
36 | P a g e
a=extmap:14/sendrecv http://protocols.cisco.com/timestamp#100us
a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42801E;packetization-mode=1;max-fs=1621;max-rcmd-nalusize=32000;level-asymmetry-allowed=1
a=imageattr:126 recv [x=[32:1:1280],y=[18:1:720],par=1.7778,q=1.00]
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42801E;packetization-mode=0;max-fs=1621;level-asymmetry-allowed=1
a=imageattr:97 recv [x=[32:1:1280],y=[18:1:720],par=1.7778,q=1.00]
a=rtpmap:111 X-ULPFECUC/90000
a=fmtp:111 max_esel=1420;m=8;max_n=32;FEC_ORDER=FEC_SRTP
a=content:main
a=rtcp-fb:* ccm fir
a=rtcp-fb:* ccm tmmbr
a=rtcp-fb:* nack pli
m=application 0 RTP/AVP 125
c=IN IP4 198.18.133.76
a=rtpmap:125 H224/4800
a=rtcp:31159
Same cycle repeats when CVP Sends an Invite to CUCM:
741741: 198.18.133.13: Sep 20 2022 11:25:02.785 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message binding info (NB):
TCP local[[ port = 0 (198.18.133.13)]] remote[[ port = 5060 (198.18.133.3) ]], Connection ID =
null, network = DEFAULT, TSIP = false
8741742: 198.18.133.13: Sep 20 2022 11:25:02.785 -0400: %_Connection-7com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: Sending Message (NB): INVITE
sip:1080@cucm1.dcloud.cisco.com;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 198.18.133.13:5060;branch=z9hG4bK6VD7zVvGnoOa4c28s+rKVQ~~148160
Max-Forwards: 68
To: <sip:1080@cucm1.dcloud.cisco.com;transport=tcp>
From: 1082 <sip:1082@198.18.133.13:5060>;tag=ds30868c5
Call-ID: 5BAE65800001000000005DCB038512C6-166368750278532168@198.18.133.13
CSeq: 1 INVITE
Content-Length: 1413
Contact: <sip:1082@198.18.133.13:5060;transport=tcp>;audio;video
37 | P a g e
Expires: 15
User-Agent: CVP 12.6 (1) ES-10 Build-50
Call-Info: <sip:198.18.133.3:5060>;purpose=x-cisco-origIP
Call-Info: <urn:x-cisco-remotecc:callinfo>;x-cisco-video-traffic-class=DESKTOP
Date: Tue, 20 Sep 2022 15:24:48 GMT
Min-SE: 1800
Allow: INVITE, OPTIONS, INFO, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE,
NOTIFY
Allow-Events: presence
Session-ID:
00007c6900105000a000005056a06cb9;remote=00000000000000000000000000000000
Cisco-Guid: 1538155904-0000065536-0000024011-0059052742
Session-Expires: 1800
P-Asserted-Identity: <sip:1082@198.18.133.3>
Remote-Party-ID: <sip:1082@198.18.133.3>;party=calling;screen=yes;privacy=off
Cisco-Gucid: 5BAE65800001000000005DCB038512C6
Supported: timer
Supported: resource-priority
Supported: replaces
Supported: X-cisco-srtp-fallback
Supported: Geolocation
Supported: sdp-anat
Content-Type: application/sdp
App-Info: <198.18.133.13:7000:7443>
v=0
o=Cisco-CVP-B2BUA 289844526 289844526 IN IP4 198.18.133.13
s=SIP Call
c=IN IP4 198.18.133.76
b=TIAS:768000
b=AS:768
t=0 0
a=cisco-mari:v1
a=cisco-mari-rate
m=audio 22894 RTP/AVP 0 8 18 104 105 114 111 101
38 | P a g e
a=extmap:14/sendrecv http://protocols.cisco.com/timestamp#100us
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:104 G7221/16000
a=fmtp:104 bitrate=32000
a=rtpmap:105 G7221/16000
a=fmtp:105 bitrate=24000
a=rtpmap:114 opus/48000/2
a=rtpmap:111 X-ULPFECUC/8000
a=fmtp:111 max_esel=1420;m=8;max_n=32;FEC_ORDER=FEC_SRTP
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
m=video 25680 RTP/AVP 126 97 111
b=TIAS:768000
a=label:11
a=extmap:14/sendrecv http://protocols.cisco.com/timestamp#100us
a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42801E;packetization-mode=1;max-fs=1621;max-rcmd-nalusize=32000;level-asymmetry-allowed=1
a=imageattr:126 recv [x=[32:1:1280],y=[18:1:720],par=1.7778,q=1.00]
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42801E;packetization-mode=0;max-fs=1621;level-asymmetryallowed=1
a=imageattr:97 recv [x=[32:1:1280],y=[18:1:720],par=1.7778,q=1.00]
a=rtpmap:111 X-ULPFECUC/90000
a=fmtp:111 max_esel=1420;m=8;max_n=32;FEC_ORDER=FEC_SRTP
a=content:main
a=rtcp-fb:* ccm fir
a=rtcp-fb:* ccm tmmbr
a=rtcp-fb:* nack pli
m=application 31158 RTP/AVP 125
a=rtpmap:125 H224/4800
39 | P a g e
Verify heartbeat and Routes Configuration:
SPOG on PCCE will have heartbeats and routes that will be pushed to CVP Call servers.
Navigate to Overview->Call Settings -> Route Settings
Click on SIP Server Groups
Verify SIP Server Group and their elements to ensure all IP addresses are in place in the groups.
Then Click on Properties and verify SIP Heartbeats parameters. Tweak if needed.
You can verify these settings on CVP under:
C:\Cisco\CVP\conf\sip.properties file
Eg:
# System configuration static routes for local routing without an outbound proxy.
SIP.System.Proxy.Route =
13*,cucm1.dcloud.cisco.com;777*,vvb1.dcloud.cisco.com;91919191*,vvb1.dcloud.cisco.com;92
929292*,vvb1.dcloud.cisco.com;
# Use heartbeating mechanism with LOCAL SRV Server Groups elements.
SIP.ServerGroupHeartbeats = true
# The number of failed heartbeats before marking the destination as unreachable.
SIP.ServerGroupHBNumTries = 3
40 | P a g e
Verify VXML Applications running on CVP VXML:
Navigate to and execute C:\Cisco\CVP\VXMLServer\admin\status.bat
This will have some information on the applications/sessions and VXML license usage:
1. You can verify VXML license usage.
2. You can verify How many applications are running and their status.
3. How many active callers are on each application and
4. How many sessions are ending.
Verify Media server health.
Navigate to C:\inetpub\logs\LogFiles\W3SVC1
Open any u_xxxxx.log file. Time will be in GMT.
Any 4xx or 5xx code is an IIS server issue.
41 | P a g e
Example of a normal media fetch:
2022-11-08 15:09:59 198.18.133.13 GET /en-us/app/Cumulus/ICMMusic_30.wav - 80 198.18.133.143 Cisco+Voice+Browser/1.0+(Cache+activated) - 304 0 0 0
This wav file has an HTTP 304 code.
It also tells you VVB tried to fetch it and its IP address so that troubleshooting efforts can be
focused on a certain VVB if needed.
CallbackServlet:
You can verify if your CVP Reporting server is configured and working fine with CCB by running
this servlet in browser. This will show you call GUID’s and status of all CCB calls.
http:// < IP/Hostname/FQDN of CVP Reporting server >:8000/cvp/CallbackServlet?method=Diag
42 | P a g e
On OAMP, it is under System -> Courtesy Callback
On SPOG, it is under Features -> Courtesy Callback
43 | P a g e
Lab 4: UCCE Certificates (CA Signed)
In this lab you will review certificates in UCCE/PCCE environment. Since introduction of Smart
licensing and security, certificates have been of utmost importance in any deployment. You will
look at some commands on UCCE servers to verify certificates. UCCE 12.6 is now using its own
keystore instead of the JDK keystore for a simplified experience.
Location: <ICM install directory>\ssl\cacerts
Relevant Topology Information
Server
IP Address
OS Login - Password
WKST1
198.18.133.75
DCLOUD\sjeffers – C1sco12345
AW
198.18.133.11
DCLOUD\Administrator – C1sco12345
PG
198.18.133.12
DCLOUD\Administrator – C1sco12345
Exercise 1: Verify Certificates on UCCE Servers
Step 1. On command prompt, type this to navigate to CCE_JAVA_HOME
cd %CCE_JAVA_HOME%\bin
Step 2. To list the existing certificates, you can use this command:
keytool.exe -list -keystore <ICM install directory>\ssl\cacerts
Default password is ‘changeit ‘unless changed by admins.
44 | P a g e
Open CCEAdmin and verify the current certificate presented by CCEAdmin and its chain:
45 | P a g e
In IIS, verify which certificate is present under Default Website Bindings:
46 | P a g e
Few points to keep in mind to verify entire certificate chain is present:

Ensure Root certificate is present

Verify Intermediate certificate is present as well if any

Ensure server certificate is present (Rogger/PG etc.)
Step 3. To delete a certificate, you can use this command:
keytool.exe -delete -alias <AW FQDN> -keystore <ICM install directory>\ssl\cacerts
Step 4. To upload a certificate, you can use this command:
keytool.exe -import -file <certificate with fully qualified path> -alias <alias
name> <AW FQDN> -keystore <ICM install directory>\ssl\cacerts
It is best to use the FQDN of the component you are uploading the certificate for. It makes it easy
to search for that certificate later.
47 | P a g e
Step 5. Since command prompt’s output could be cryptic to use when viewing certificate
information, you can export entire list to a text file and then use any text editor to view it.
keytool.exe -list -keystore <ICM install directory>\ssl\cacerts > c:\certs.txt
Lab 5: Troubleshoot Smart Licensing.
In this lab you will check logs related to smart licensing on CCE and go through some messages
related to the feature.
Relevant Topology Information
Server
IP Address
OS Login - Password
WKST1
198.18.133.75
DCLOUD\sjeffers – C1sco12345
AW
198.18.133.11
DCLOUD\Administrator – C1sco12345
CVP
198.18.133.13
DCLOUD\Administrator – C1sco12345
Exercise 1: Login to WKST1 and RDP to AW
Step 1. Navigate to C:\icm\tomcat\webapps\unifiedconfig\WEB-INF\classes\conf and edit
log4j.xml
48 | P a g e
Enable these debugs to generate debug messages in logs. Disable them after troubleshooting:
<logger name="com.cisco.ccbu.smartagentmanager" additivity="false">
<level value="debug" />
<appender-ref ref="smartAgentAppender" />
</logger>
<logger name="com.cisco.nesla" additivity="false">
<level value="debug" />
<appender-ref ref="smartAgentAppender" />
</logger>
You will have to restart Cisco Tomcat for changes to get into effect.
For CVP, it is located in:
C:\Cisco\CVP\conf\log4j_wsm.xml
Step 2. Collect logs from these location to review.
From Logger :
CCBU logs, Error Logs, Smart Agent logs under C:\icm\tomcat\logs
From Distributor :
CCBU logs, Error Logs under C:\icm\tomcat\logs
From Call/VXML Server :
CVP logs, Error logs, Smart Agent logs under C:\Cisco\CVP\logs\WSM .
49 | P a g e
Step 3. Verify “.sl_truststore” has data in it. Especially state.yml:
•
ios_core.p7b : Cisco Official Cert P7B file ---- Certificate pool
•
CiscoRootCAM1.cer : Cisco Official M1 file used to verify the P7B file:
•
db.properties : DB file used by TP module to store some db data.
•
call_home_ca : Contain the all PEMs(Privacy Enhanced Mail ) which extracted from P7B file.
•
State.yml : This file stores the Last registration information.
•
agentCerts : Stores the certificate for Smart Agent ( Contains root,CA, sub CA certificate etc )
•
agentStore : This is like a keystore for the Smart Agent.
Truststore on CVP and ICM:
Open state.yml and check Registration and Authorization states.
50 | P a g e
Lab 6: Troubleshoot CUIC and Live Data
In this lab you will troubleshoot LiveData from various angles and try to isolate it to a where the
issue is.
Following outputs are from a separate lab however you can still run all the commands on LiveData
Publisher.
Relevant Topology Information
Server
IP Address
OS Login - Password
WKST1
198.18.133.75
DCLOUD\sjeffers – C1sco12345
ROGGER
198.18.135.37
DCLOUD\Administrator – C1sco12345
AW
198.18.133.11
DCLOUD\Administrator – C1sco12345
PG
198.18.133.12
DCLOUD\Administrator – C1sco12345
CUIC
198.18.133.15
administrator – dCloud!23
Exercise 1: Login to WKST1 and RDP to ROGGER, PG and CUIC
Step 1. Verify LiveData and Rogger/PG connectivity over TIP/TOS. Active side of LiveData will
connect to TIP and TOS however inactive side of LiveData will only connect to TOS.
netstat -ano | findstr <IP of LD Pub or Sub>
ROGGER:
PG:
51 | P a g e
LiveData:
show open ports regexp 4[0-5]03[45]
[For Instance-0]
show open ports regexp 4[0-5]07[45]
[For Instance-1]
Step 2. Important LiveData commands.
show live-data failover
Possible States:
PAIRED-ACTIVE – Cluster is working fine and this is the LiveData Active Server.
PAIRED-INACTIVE – Cluster is working fine and this is the LiveData Inactive Server.
ISOLATED-ACTIVE – Lost connectivity to another server or there is no HA and is the active LD
server.
ISOLATED-INACTIVE – Lost connectivity to another server or there is no HA and is the inactive LD
server.
UNAVAILABLE – LiveData did not deploy properly.
Out of Service – Live Data deployed however there are some configurations and/or connectivity
issues.
52 | P a g e
show live-data cuic-datasource <cuic-addr> <cuic-port> <cuic-user>
show live-data aw-access – Must succeed in AW connectivity.
show socketio status – This must be active on Pub and Sub.
utils ntp status – NTP must be in sync
Step 3. Collect Live Data and CUIC Logs along with important outputs
53 | P a g e
From LiveData:
file get activelog livedata/logs recurs compress
file get activelog syslog recurs compress
file get install system-history.log
From CUIC:
file get activelog cuic/logs/cuic recurs compress
file get activelog cuic/logs/cuicsrvr recurs compress
From AW:
select * from Machine_Host
select * from Machine_Service
select * from Machine_Address
Lab 7: Troubleshoot Finesse
In this lab you will troubleshoot Finesse and verify various aspects of it.
Step 1. RDP to Workstation 1 (198.18.133.75) – User: dcloud\sjeffers, Password: C1sco12345 or by
clicking ‘Remote Desktop’
Step 2. On the desktop double-click on mRemoteNG
Step 3. On mRemoteNG, open a connection to Finesse by clicking on
Exercise 1: Verify current logging levels
Step 1. Execute command ”utils finesse log configuration list ” to check the current logging levels
in the server.
54 | P a g e
Step 2. Execute command “utils finesse log configuration add ?” to get the syntax:
Step 3. Add logging for webservices com.cisco.cc.common.subsystem level ERROR
utils finesse log configuration add webservices com.cisco.cc.common.subsystem ERROR and hit
ENTER.
Step 4. Re-run ”utils finesse log configuration list ” to confirm the added trace level on
webservices module.
55 | P a g e
Step 5. Update webservices com.cisco.cc.common.subsystem level to DEBUG
utils finesse log configuration update webservices com.cisco.cc.common.subsystem DEBUG and
hit ENTER.
Step 6. Re-run ”utils finesse log configuration list ” to confirm the updated trace level on
webservices module.
Step 7. Disable the webservices com.cisco.cc.common.subsystem trace by executing the
command.
utils finesse log configuration delete webservices com.cisco.cc.common.subsystem
56 | P a g e
Step 8. Again run ”utils finesse log configuration list ” to make sure we disabled trace level on
webservices module.
Exercise 2: Collect Finesse Logs via CLI or RTMT or Web
On WKST1 make sure mini sftp server is in started state.
Step 1. Open Mini sftp server, by searching the application and name and double click on it.
57 | P a g e
Step 2. If you find ‘Start’ button, It’s not started yet. Click ‘Start’ to start it.
User: administrator
Password: C1sco12345
Port: 22
Root path: C:\Users\sjeffers\Downloads
If Stop button is there, it’s started already.
Step 3. Run “file get activelog /desktop/logs/webservices” command to pull webservices log
First it will list the logs in the folder and size of the folder. Hit ‘y’ to proceed.
58 | P a g e
Step 4. When asks for SFTP server IP: Key in WKST1 server IP: 198.18.133.75
Port: 22
User ID: administrator
Password: C1sco12345
Download directory: /
Type ‘yes’ to proceed
Step 5. Make sure the logs are collected in the root directory path of the SFTP server.
59 | P a g e
Step 6. To install RTMT, open a browser from WKST1 and access the url
https://198.18.133.16:8443/plugins/CcmServRtmtPlugin.exe
Step 7. Run the application from download directory as ‘Run as administrator’
Step 8. Run the simple wizard to complete the installation, once Finished RTMT tool will be
available on the desktop
Step 9. Type in Finesse IP: 198.18.133.16 and click ‘Ok’
Step 10.
Username: administrator
Password: C1sco12345
60 | P a g e
Step 11. Click ‘Trace & Log Central’ on the left plane and then click ‘Collect Files’
Step 12. Select the required logs to be collected and click next.
61 | P a g e
Step 13. Select timeframe, Time zone, download directory details and Click ‘Finish’ to download
the logs
Step 14. Open the Download File Directory path mentioned above and confirm the logs are
downloaded.
62 | P a g e
Step 15. To Collect logs from Browser. Open a browser, type in url :
https://198.18.133.16:8445/finesse/logs
Username: administrator
Password: C1sco12345
Step 16. Click the required module logs to be downloaded.
63 | P a g e
Step 17. Click the file/folder to collect the log.
Log format  .log file, Click the file and copy and content, Save it in notepad.
Log format  .zip, Just a click will download the file to the browser downloads.
Exercise 3: Finesse PG Failover
Step 1. From WKST1, mRemote – connect PG server
Step 2. Open CCE service Control. From desktop
Step 3. Select CG3A – Ctisvr service and Click Cycle.
Step 4. Once the service is back to running. Collect Finesse webservice logs by any of the
methods explained.
64 | P a g e
Follow the events to understand
ACMI_STATE_CHANGE
Exercise 4: Fun with Finesse API’s
Step 1. Open a browser from WKST1 and run the following Finesse APIs one by one and
understand the info from the server.
https://198.18.133.16:8445/finesse/api/RuntimeConfigInfo
Username: administrator
Password: C1sco12345
Step 2. Cross check the output from the CLI access with – utils finesse node_statistics list. Make
a note of the Agent login count in the node, max skill groups per agent count, Average Skill
groups per agent count.
65 | P a g e
Step 3. Make a note of the Deployment Type, Status , Peripheral ID, Host, HTTPS and XMPP
ports, host IP.
https:// 198.18.133.16:8445/finesse/api/SystemInfo
Step 4. Make a note of Teams, Users, Supervisors, Version Info and total Skill groups
https://198.18.133.16:8445/finesse/api/ConfigInfo
66 | P a g e
Step 5. Have a look on Configured ECC variables in the system.
https://198.18.133.16:8445/finesse/api/ECCVariableConfig
Step 6. Since the lab is simplex, secondary node is blank.
https://198.18.133.16:8445/finesse/api/ClusterConfig
67 | P a g e
Step 7. Check reason codes. Change the reason code ID as 1, 2,3 …
https://198.18.133.16:8445/finesse/api/ReasonCode/1
Understand the output details like category, label etc.
68 | P a g e
Step 8. Have a look on Team names and IDs
https://198.18.133.16:8445/finesse/api/Teams
Step 9. To check individual team, put team ID at the end.
https://198.18.133.16:8445/finesse/api/Team/5021
69 | P a g e
Step 10. Understand the configured SystemVariables.
https://198.18.133.16:8445/finesse/api/SystemVariables
Exercise 4: Few Finesse CLI Commands
Step 1. From WKST1, open an SSH session to Finesse
Run utils finesse cors ? to see the available commands and then run utils finesse cors status to
know the current configuration.
.
70 | P a g e
Step 2. Run command - utils finesse maintenance initiate
As it’s a simplex setup, understand the output as there is no alternate finesse side existing.
Step 3. Run command – utils finesse show_connected_users summary
Understand number of connected user, Desktop, FIPPA and third-party users details
Lab 8: Troubleshoot Router/PG and Collect Logs
In this lab you will check RTTEST utility from Router and collect Router and PG logs.
Exercise 1: Login and execute RTTEST and dumplog
Step 1. RDP to Workstation 1 (198.18.133.75) – User: dcloud\sjeffers, Password: C1sco12345 or
by clicking ‘Remote Desktop’
71 | P a g e
Step 2. On the desktop double-click on mRemoteNG
Step 3. On mRemoteNG, open a connection to Router by clicking on
Step 4: Launch run prompt and enter cmd and click ok
Step 5. In the command prompt type rttest /cust pcce /node RouterA and hit Enter
Step 6: in the rttest prompt enter status and hit enter.
72 | P a g e
Step 7: Status of each router process is displayed
Exercise 2: How to collect router process log using Diagnostic framework portico?
Step 1: Double click Unified CCE Tools on Router desktop
Step 2: Double click Diagnostic Framework Portico
73 | P a g e
Step 3: In the login prompt provide username as administrator@dcloud.cisco.com and password
C1sco12345 and hit ‘Log In’
Step 4: From left pane under Trace click on ListTraceFiles
Step 5: In the ListTraceFiles drop down menu select Router A/rtr component and provide the date
and time range and hit ‘Submit’.
Step 6: Click on the download link to save the log file to destination you want to store.
Step 7: Repeat step 5-6 to download other component (CCAG,MDS) log
74 | P a g e
Exercise 3: How to collect pg process log using Diagnostic framework portico?
Step 1. RDP to Workstation 1 (198.18.133.75) – User: dcloud\sjeffers, Password: C1sco12345 or by
clicking ‘Remote Desktop’
Step 2. On the desktop double-click on mRemoteNG
Step 3. On mRemoteNG, open a connection to PG by clicking on
Step 4: Double click Unified CCE Tools on PG desktop
Step 5: Double click Diagnostic Framework Portico
Step 6: In the login prompt provide username as administrator@dcloud.cisco.com and password
C1sco12345 and hit ‘Log In’
75 | P a g e
Step 7: From left pane under Trace click on ListTraceFiles
Step 8: Under the ListTraceFiles , drop down menu select Peripheral Gateway 1A/mds component
and provide the date and time range and hit ‘Submit’.
Step 9: Click on the download link to save the log file to destination you want store.
Step 10: Repeat step 8-9 to download other component (PGAG, MDS,PIM,OPC,CTISVR) log
Lab 9: Troubleshoot VVB
In this lab you will troubleshoot VVB Server and check couple of important CLI commands.
Exercise 1: CLI Commands
Step 1. RDP to Workstation 1 (198.18.133.75) – User: dcloud\sjeffers, Password: C1sco12345 or
by clicking ‘Remote Desktop’
76 | P a g e
Step 2. On the desktop double-click on mRemoteNG
Step 3. On mRemoteNG, open a connection to VVB by clicking on
Step 4: Accept VVB security certificate by clicking Yes.
Step 5: Run ‘show vvb host-to-ip’ command
Shows the user entries from /etc/hosts file.
Step 6: show vvb call active voice summary
77 | P a g e
This command shows active voice call summary.
Step 7: show vvb cache cache_entries
This command shows all or selected entries that are cached.
Command syntax
show vvb cache cache_entries <start_index> <end_index>
Max cache size configured in
Max file size configured in
Step 8: show vvb cache cache_entry <URL>
This command shows details, such as size and age, of a cache entry.
Command syntax
show vvb cache cache_entry <URL>
78 | P a g e
Step 9: set vvb cache browser_cache_size
This command sets the cache size in KB. Setting cache size to 0 disables the cache. Disabling
cache does not add new entries to the cache. However, existing cache entries can be reused until
they are expired.
Command syntax
set vvb cache browser_cache_size [size_in_KB]
Step 10: set vvb cache max_file_size
This command sets the cache size in KB. Setting cache size to 0 disables the cache. That means
that new entries will not be added to the cache; however, existing cache entries will be reused
until they are expired.
Command syntax
set vvb cache max_file_size [size_in_KB]
Step 11: set vvb cache enable_browser_cache
This command enables or disables the browser cache where True is enabled and False is
disabled.
Command syntax
set vvb cache enable_browser_cache [True|False]
79 | P a g e
Step 12: set vvb cache stale_cache_entry <URL>
This command marks stale for the given cache entry URL. The stale cache entry resource gets
downloaded only for the first instance after it is marked as stale.
Command syntax
set vvb cache stale_cache_entry <URL>
Stale flag is set to true.
Step 13: set vvb cache stale_cache_entries
This command marks stale for all cache entries. The stale cache entries get downloaded only for
the first instance after it is marked as stale.
Command syntax
set vvb cache stale_cache_entries
# sign indicates file is set
80 | P a g e
Exercise 2: VVB Logs
Step 1. Open Mini sftp server, by searching the application and name and double click on it.
Step 2. If you find ‘Start’ button, it’s not started yet. Click ‘Start’ to start it.
User: administrator
Password: C1sco12345
Port: 22
Root path: C:\Users\sjeffers\Downloads
If Stop button is there, it’s started already.
81 | P a g e
Step 1. RDP to Workstation 1 (198.18.133.75) – User: dcloud\sjeffers, Password: C1sco12345 or
by clicking ‘Remote Desktop’
Step 2. On the desktop double-click on mRemoteNG
Step 3. On mRemoteNG, open a connection to VVB by clicking on
Step 4. Once VVB is connected, execute command “file get activelog /uccx/log/MIVR” command
to pull Engine or MIVR log
First it will list the logs in the folder and size of the folder. Hit ‘y’ to proceed
82 | P a g e
Step 5. When asks for SFTP server IP: Key in WKST1 server IP: 198.18.133.75
Port: 22
User ID: administrator
Password: C1sco12345
Download directory: /
Type ‘yes’ to proceed
Step 6. Make sure the logs are collected in the root directory path of the SFTP server.
Via RTMT
Step 1. Open a browser from WKST1 and access the url
https://198.18.133.143/plugins/CcmServRtmtPlugin.exe
83 | P a g e
Step 2. Run the application from download directory as ‘Run as administrator’
Step 3. Run the simple wizard to complete the installation, once Finished RTMT tool will be
available on the desktop
Step 4. Type in VVB IP: 198.18.133.143 and click ‘Ok’
Step 5.
Username: administrator
Password: dCloud!23
Step 6. Click ‘Trace & Log Central’ on the left plane and then click ‘Collect Files’
84 | P a g e
Step 7. Select the required logs to be collected and click next.
Step 8. Select timeframe, Time zone, download directory details and Click ‘Finish’ to download
the logs
85 | P a g e
Step 9. Open the Download File Directory path mentioned above and confirm the logs are
downloaded.
86 | P a g e