Unity Traces & Troubleshooting Michael Hollowell CCIE #4114 (Oct 4, 1998) UCBU – Senior Escalation Engineer rev: 06/06/11 Agenda • The playing field. • Setting the traces. • Customizing the traces. • Gathering everything up. • Diagnotistic Examples • Q&A. © 2011 Cisco Systems, Inc. All rights reserved. 2 On-line Troubleshooting Guides Unity trouble shooting Guides http://www.cisco.com/en/US/products/sw/voicesw/ps2237/prod_troubleshooting_guides_list.html 5.0 Trouble shooting Guide http://www.cisco.com/en/US/docs/voice_ip_comm/unity/5x/troubleshooting/guide/ex/5xcutsgex.html © 2011 Cisco Systems, Inc. All rights reserved. 3 The Playing Field – Ways to Trouble shoot • Windows System / Application Event Log • Diagnostic Logging via the Unity Diagnostic Tool (UDT) • Port Status Monitor (2.0+) • GUSI (2.0 +) • Low Level Debugging • Miscellaneous Logs © 2011 Cisco Systems, Inc. All rights reserved. 4 System / Application Log – Gathering Apps • Eventvwr.exe -> Save Log File As... - .evt (GMT adjusted time) - .csv (current server time) • GUSI (Gather Unity System Info. 2.x+) • UDT -> Gather Standard Logs - AvCsMGr, App, & Sys Logs Select Logs* Over a specified Time Frame *Option to export App & Sys to CSV is there, by default it is not included © 2011 Cisco Systems, Inc. All rights reserved. 5 Reading System/Application Logs • EVT - eventvwr -> open log file… Should be opened on a Unity server of the same version in order to decode events properly.* - Unity Diagnostic Viewer • CSV - Excel, Notepad, Word, etc. * timestamps might not match Diags © 2011 Cisco Systems, Inc. All rights reserved. 6 UDT Traces - Gathering Correct way: Using UDT –> Gather log Files Incorrect Way: Taking them dircectly from the \commserver\logs directory unless otherwise specified. © 2011 Cisco Systems, Inc. All rights reserved. 7 This is your log… This is your log formatted Plucked from the \commserver\logs directory: 16:38:52:094,AvDiagnostics_MC,1225,3760,-1,,Notifier,24,John Doe,0,Text(0x1),Urgent Only 16:38:52:093,AvDiagnostics_MC,1221,5784,1,,Notifier,20,0x5e90001,32,Any 16:38:52:125,AvDiagnostics_MC,1258,3760,1,,Notifier,16,Queued,John Doe.jdoe@local.com Gathered from the UDT*: 16:38:52:094 (AvDiagnostics_MC,1225,Notifier,24) [Thread 3760] John Doe has 0 messages of type Text(0x1) Urgent Only 16:38:52:093 (AvDiagnostics_MC,1221,Notifier,20) [Thread 5784] Ticket 0x5e90001 granted for resource 32 (requested Any) 16:38:52:125 (AvDiagnostics_MC,1258,Notifier,16) [Thread 3760] Queued SMTP task for mailuser=John Doe, addr#=jdoe@local.com. * You can format the unformatted logs by placing them into the \commserver\logs directory of the same Unity version from which the logs were pulled from and then use UDT to gather/format them. © 2011 Cisco Systems, Inc. All rights reserved. 8 Port Status Monitor (2.x +) • Replaced the Original Port Status Monitor (PSM) • Allows you to monitor the voiceports in real time • Unity version 4.2 and later • Latest version 2.0.2.4 • Easiest tool for watching MWI activity and User interaction. • Training Video: ftp://ftpeng.cisco.com/lindborg/Unity42/NEWPSM.exe © 2011 Cisco Systems, Inc. All rights reserved. 9 GUSI (2.0 +) • Gather Unity System Information • Current Version 2.0.4 • Key New Feature within 2.x - Configuration Conformance Checking • Always check with ciscounitytools.com for latest version • Online help file: http://ciscounitytools.com/Applications/Unity/GUSI/Help/GatherUnitySystemInfo.htm © 2011 Cisco Systems, Inc. All rights reserved. 10 Debugging Advanced Use Only Windbg.exe - May require Unity, Microsoft, or other 3rd party Symbols - Best to get a memory dump Use “!analyze –v” dbgview.exe - VMO client issues © 2011 Cisco Systems, Inc. All rights reserved. 11 Miscellaneous Logs \<os install>\system32\logfiles\... …\w3svc1\ex<#>.log …\HTTPERR\httperr1.log (www logs enable by default : PCA, SAweb) …\smtpsvc1 (VPIM, Bridge, Text Paging, etc.) (must enable first on the protocol) \<os install>\KB*.log (list of all KBs installed on the system) (can also use add/remove programs) © 2011 Cisco Systems, Inc. All rights reserved. 12 Agenda • The playing field. • Setting the traces. • Customizing the traces. • Gathering everything up. • Diagnotistic Examples • Q&A. © 2011 Cisco Systems, Inc. All rights reserved. 13 Unity Architecture © 2011 Cisco Systems, Inc. All rights reserved. 14 Unity Architecture For More Information … A lot more detail about the architecture diagram, with information about the Unity components and processes in it, can be found in these resources: Unity Architecture Overview 4.0(3) Cisco Unity Deployment and Solutions Guide We’re not going to go into quite that level of detail in this presentation, but we’ll do a couple of process walkthroughs to get an idea of how to understand and apply the architecture diagram. © 2011 Cisco Systems, Inc. All rights reserved. 15 The Complexity of Unity Diags Multiple Log Managers AvCsMgr process Log Manager diag_AvCsMgr_20060126_220003.txt AvDSAD process Log Manager diag_AvDSAD_20060126_220003.txt Each Unity process runs its own copy of the log manager, and creates its own diagnostic file. © 2011 Cisco Systems, Inc. All rights reserved. 16 The Complexity of Unity Diags Component vs. Process Example AvCsMgr process MIU Component Log Manager diag_AvCsMgr_20060126_220003.txt Example: Gathering MIU Traces 1. Set MIU Traces in Unity Diagnostic Tool 2. Reproduce Behavior 3. Use UDT to gather AvCsMgr diag file, because the MIU component lives in the AvCsMgr process © 2011 Cisco Systems, Inc. All rights reserved. 17 The Complexity of Unity Diags Same Component, Different Processes AvCsMgr process DOH component CsBridgeConnector process DOH component AvUmrSyncSvr process DOH component © 2011 Cisco Systems, Inc. All rights reserved. Some components run under more than one process. For example, you can find output from DOH traces in diag files from each of several Unity processes. 18 The Complexity of Unity Diags Some Diag Files Are Outside the UDT UDT AvCsMgr process Log Manager Unity Setup © 2011 Cisco Systems, Inc. All rights reserved. diag_AvCsMgr_20060126_220003.txt tempu.log PWResults.html OsqlDump_ 19 \Commserver\Log File Naming Convention diag_AvCsMgr_20060126_220003.txt Diag = Diagnostic type AvCsMgr = LogMgr owning Process 20060126 = Date using YYYYMMDD 220003 = Time using HHMMSS Where the Date/Time is when the log file was CREATED. © 2011 Cisco Systems, Inc. All rights reserved. 20 \commserver\log Data Files Take a good look at the Commserver\Logs directory, and you’ll notice some files obviously related to Unity components, that start with “data.” These files are actually used to generate Unity reports, and are not a core part of the Unity diag functionality. They’re worth mentioning here, because people sometimes mistakenly gather them directly from the Commserver\Logs directory, thinking they are Unity diags. © 2011 Cisco Systems, Inc. All rights reserved. 21 UDT – MACRO / MICRO MACRO (as of 5.0.1) AMIS Problems AMIS Message Traces Delays in Answering Calls Call Flow Diagnostics Conversation State Traces Bridge Directory Sync Traces VPIM Directory Sync Traces Directory Monitor Traces Call Control (Miu) Traces Traces for MWI Problems Traces for other notification problems Skinny TSP traces Unity Startup Media (Wave) Traces 1 - High Media (Wave) Traces 2 - Medium Media (Wave) Traces 3 - Low Secure Messaging Traces Text to Speech (TTS) Traces MICRO (most components broken out) © 2011 Cisco Systems, Inc. All rights reserved. 22 Setting the Traces : Default Recommendations • Look for a relevant MACRO trace • Always include : - Conversation State Traces - Call Flow Diagnostics • Additional Traces to focus on specific problem : - AvRdbSvr 11 (SQL query Generation) - DOH 10+ & MALEx 10+ (Exchange Mailbox issues) - notescommon 10+ & MALLn 10+ (Domino Issues) - AvSaDbConn 10+ (SAWeb add/modify/delete issues) - NodeMgr 10+ (Failover issues) DSAD 10+ (Local Unity Objects to/from AD Sync) DSGlobalCatalog 10+ (Object not local to Unity to/from AD sync) DSDomino 10+ (Unity/from to Domino Sync) © 2011 Cisco Systems, Inc. All rights reserved. 23 Agenda • The playing field. • Setting the traces. • Customizing the traces. • Gathering everything up. • Diagnotistic Examples • Q&A. © 2011 Cisco Systems, Inc. All rights reserved. 24 Customizing Traces – Personal Macro • Place/Review them at : \commserver\utilities\unitydiagnostictool\udtxml\ • Filename MUST start with H. • Filename MUST have an .xml extension. • File contents MUST be in XML format. • Example* : HTTAC-PhoneView.xml *Complete PhoneView Trace in Notes Section © 2011 Cisco Systems, Inc. All rights reserved. 25 Customized Macro Trace - Example <?xml version="1.0"?> <GA NAME="Phone View"> <DESCRIPTION>These traces help you troubleshoot Cisco Unity Phone View Issues.</DESCRIPTION> <TRACE NAME="For TAPI and SIP"> <DESCRIPTION>These traces help you troubleshoot Cisco Unity Phone View Issues.</DESCRIPTION> <DIAG GROUP="CDE"> <FLAG ID="01" /> <FLAG ID="20" /> <FLAG ID="21" /> <FLAG ID="22" /> <FLAG ID="23" /> </DIAG> </TRACE> </GA> © 2011 Cisco Systems, Inc. All rights reserved. 26 Agenda • The playing field. • Setting the traces. • Customizing the traces. • Gathering everything up. • Diagnotistic Examples • Q&A. © 2011 Cisco Systems, Inc. All rights reserved. 27 UDT – Gathering Logs • If not sure, gather all over a time frame - Note Time on the Unity Server - Recreate the Issue - Set start time to be the time from above - Gather for the default 1 hour • Preferred : - Press Start New Log File - Recreate the Issue - Press Start New Log File - Gather ‘Select Logs’ grab the second one to bottom © 2011 Cisco Systems, Inc. All rights reserved. 28 Addition Information to include • GUSI (Grab latest version available on unitytools.com) • Unity Time when the issue was occurring. • Affected Subscriber, Distribution lists, DC/GC Server, Exchange/Domino Servers, Port the call arrived on, etc. • Background information (step by step of what was recreated) • Working* vs. Non-working Trace *working can be a lab recreate using your own server mocking up the customers environment if working is not available on the customers server. © 2011 Cisco Systems, Inc. All rights reserved. 29 Quirky things to gather on the Unity Server • Skinny shows up in SvcHost starting Unity 4.x and later. • Permission Wizard: <logged in acct>\temp\PWResults.html • Unexpected Notes terminations: <lotus notes>\Data\IBM_Technical_Support (zip up entire directory) © 2011 Cisco Systems, Inc. All rights reserved. 30 Personal Communication Assistant • Setup \<logged in account>\temp\cscoserv_*.log\ • Application Logging \CommServer\Cscoserv\Tomcat\Logs\ciscopca*.txt \<OS install>\system32\logfiles\w3svc1\ex<#>.log © 2011 Cisco Systems, Inc. All rights reserved. 31 Unity Upgrade Failures • 5.x and later \commserver\logs\tempu_<date/time>.log \<logged in account>\temp\syscheck.log \commserver\logs\osqldump_<date/time>.txt \commserver\logs\sqlsync_<#>.txt \commserver\logs\cfgCUApp_<date/time>.txt \Commserver\Logs\diag_Install_<date/time>.txt • 4.x and earlier \<logged in account>\temp\tempu.log \<logged in account>\temp\syscheck.log \commserver\logs\osqldump_<date/time>.txt \commserver\logs\sqlsync_<#>.txt \commserver\logs\cfgCUApp_<date/time>.txt \Commserver\Logs\diag_Install_<date/time>.txt © 2011 Cisco Systems, Inc. All rights reserved. 32 Exchange Server – Miscellaneous Logs • Application and System Event Logs • Message Tracking (if delivery related – must be enabled) - <Exchange Server Path>\Web.log • SMTP Logs (if routing off the local exchange server – must be enabled) • IVC logs (Voice Connector – need to bump up the level) - <Exchange Server Path>\VoiceGateway\logFiles - GwIvc_<date>.log (diags) - GwIvc_perf_<date>.log (perfs) © 2011 Cisco Systems, Inc. All rights reserved. 33 Domino – Miscellaneous Logs • Log.nsf - Tip : Create a blank NSF based on the log.ntf template and cut/paste the relevant timeframe into it. - Has to be viewed via Notes client - 3 Categories : Mail Routing, Replication, & Misc. Events • Message Tracking - <domino>\data\mtdata\mtstore.nsf - Has to be viewed via Notes client © 2011 Cisco Systems, Inc. All rights reserved. 34 Agenda • The playing field. • Setting the traces. • Customizing the traces. • Gathering everything up. • Diagnotistic Examples • Q&A. © 2011 Cisco Systems, Inc. All rights reserved. 35 Some General Techniques for Reading Diag Files Use Find to look for strings seen elsewhere Use Findstr to filter Working vs. Non-working Pay attention to timing Beware error reports by themselves Assemble a story out the diag lines. © 2011 Cisco Systems, Inc. All rights reserved. 36 Diagnostic Example – Basic Call Flow Background: - Caller 8015 leaves a voicemail for 8005 Traces Enabled: (All found within UDT Macro) - All Call Flow Diag - Conv State Traces - Call Control (Miu) Traces - Traces for MWI problems - SKinny TSP Traces Traces Gathered: - AvCsMgr (diag_AvCsMgr_20110106_220000_fmtd.txt) - SvcHost (diag_svchost_20110106_220000_fmtd.txt) © 2011 Cisco Systems, Inc. All rights reserved. 37 Diagnostic Example – Basic Call Flow Unity Diagnostic Tool - Found in Tools Depot on the Unity Sever © 2011 Cisco Systems, Inc. All rights reserved. 38 Diagnostic Example – Basic Call Flow Last File listed within each tree is the most current diagnostic file © 2011 Cisco Systems, Inc. All rights reserved. 39 Diagnostic Example – Basic Call Flow Navigating the diag Files - Start backwards search starting from the bottom of the file (since this is the most current information) - Our example, a voicemail is being left for Extension 8005, so we will search for “8005” 10:59:25:772 **NOT FORMATTED**,AvDiagnostics_MC,162,4156,-1,, SkinnyTSP,12,[Thread 0x0000103C] [Device 6] StationCallInfoV2Message: Extracted CallerID='8015' () CalledID='8005' () RedirectReason=2=RfrCallFwdRNA LastRedirectingID='8005' () LastRedirectReason=2=RfrCallFwdRNA - To isolate this call only, we can use the [Device 6] identifier © 2011 Cisco Systems, Inc. All rights reserved. 40 Diagnostic Example – Basic Call Flow To extract this one call flow use “Wingrep” or windows “find” Example (from the windows CLI): find "[Device 6]" .\diag_svchost_20110106_220000_fmtd.txt > .\call.txt This will parse out all occurrences of [Device 6] and place them into a file named “call.txt” © 2011 Cisco Systems, Inc. All rights reserved. 41 Diagnostic Example – Basic Call Flow 10:59:25:771 **NOT FORMATTED**,AvDiagnostics_MC,162,4156,-1,,SkinnyTSP,12,[Thread 0x0000103C] [Device 6] Receive StationSetRingerMessage (20 bytes) ringMode=2=StationInsideRing ringDuration=1=StationNormalRing lineInstance=1 callReference=26043776 … 10:59:25:771 **NOT FORMATTED**,AvDiagnostics_MC,162,4156,-1,,SkinnyTSP,24,[Thread 0x0000103C] [Device 6] CAvSkinnyCallStatus::SignalCallState: callState=4=TsRingIn callReference=26043776 … 10:59:25:959 **NOT FORMATTED**,AvDiagnostics_MC,162,4156,-1,,SkinnyTSP,24,[Thread 0x0000103C] [Device 6] CAvSkinnyCallStatus::SignalCallState: callState=1=TsOffHook callReference=26043776 … 10:59:25:960 **NOT FORMATTED**,AvDiagnostics_MC,162,4156,-1,,SkinnyTSP,24,[Thread 0x0000103C] [Device 6] CAvSkinnyCallStatus::SignalCallState: callState=5=TsConnected callReference=26043776 … 10:59:52:162 **NOT FORMATTED**,AvDiagnostics_MC,162,4156,-1,,SkinnyTSP,12,[Thread 0x0000103C] [Device 6] Receive StationCallStateMessage (28 bytes) callState=2=TsOnHook lineInstance=1 callReference=26043776 privacy=0=PrivacyNone precedence=[precedenceLv=4 precedenceDm=0] Callstates : 4=TsRingIn -> 1=TsOffHook -> 5=TsConnected -> 2=TsOnHook © 2011 Cisco Systems, Inc. All rights reserved. 42 Diagnostic Example – Basic Call Flow To find various call flows within AvCsMgr Example find " CallInfo" .\diag_AvCsMgr_20110106_220000_fmtd.txt > callinfo.txt (Note there is a space after the first double quote before the callinfo word) 10:58:50:460 (AvDiagnostics_MC,2195,MiuGeneral,12) [Thread 4212] [Port 1] [B259F01316B443F58D74C607FDFF2976] [Thread 0x00001074] [Port 1] INTEGRATION CallInfo received (Origin Internal | Reason FwdNoAnswer | CallerID 8015 | CalledID 8005 | RedirectingID 8005). 10:59:25:772 (AvDiagnostics_MC,2195,MiuGeneral,12) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] [Thread 0x0000107C] [Port 1] INTEGRATION CallInfo received (Origin Internal | Reason FwdNoAnswer | CallerID 8015 | CalledID 8005 | RedirectingID 8005). Since only two calls were found we will have to isolate the call in question via the time the call came in found via monitoring PSM or through SVCHost diag (in this case - 10:59:25 time frame). *** To isolate, use the Call GUID : [8985A5D73D924EFABBE5CEAB14E778BA] © 2011 Cisco Systems, Inc. All rights reserved. 43 Diagnostic Example – Basic Call Flow Using the unique call guid, we can isolate a certain call within the AvCsMgr Diag: Example: find “[8985A5D73D924EFABBE5CEAB14E778BA]” .\diag_AvCsMgr_20110106_220000_fmtd.txt > callguid.txt © 2011 Cisco Systems, Inc. All rights reserved. 44 Diagnostic Example – Basic Call Flow Call routing properly 10:59:25:772 (AvDiagnostics_MC,642,RulerDomain,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] CRulerDomain::FindFirstTrueRule: StartIndex=[0] 10:59:25:771 (AvDiagnostics_MC,1316,RulerDomain,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Evaluating Rule 'Attempt Forward to Greeting' in Domain 'Routing'. 10:59:25:772 (AvDiagnostics_MC,1319,RulerDomain,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Expression 'Reason IN 2 4 8 64' evaluates to true. 10:59:25:771 (AvDiagnostics_MC,1317,RulerDomain,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Rule 'Attempt Forward to Greeting' in Domain 'Routing' evaluated to TRUE. 10:59:25:772 (AvDiagnostics_MC,642,RulerDomain,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] CRulerDomain::FindFirstTrueRule: Rule evaluates to TRUE; Index=[1] 10:59:25:771 (AvDiagnostics_MC,1325,RulerDomain,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] First Rule to evaluate to TRUE in Domain 'Routing' is 'Attempt Forward to Greeting': Action='AttemptForward'. Call Routing is decided upon based on whether the call was: - “Direct” call into Unity - “Forwarded” call into Unity “Attempt Forward to Greeting” is one of the Unity’s default “Forwarded” rules © 2011 Cisco Systems, Inc. All rights reserved. 45 Diagnostic Example – Basic Call Flow Bad Forwarded routing rule 12:05:40:055 (AvDiagnostics_MC,642,RulerDomain,11) [Thread 4220] [Port 1] [F12392F48B7E4A55AFDCEE298462574A] CRulerDomain::FindFirstTrueRule: StartIndex=[0] 12:05:40:056 (AvDiagnostics_MC,1316,RulerDomain,11) [Thread 4220] [Port 1] [F12392F48B7E4A55AFDCEE298462574A] Evaluating Rule 'New-Rule' in Domain 'Routing'. 12:05:40:055 (AvDiagnostics_MC,1319,RulerDomain,11) [Thread 4220] [Port 1] [F12392F48B7E4A55AFDCEE298462574A] Expression 'DialedNumber == 8005' evaluates to true. 12:05:40:056 (AvDiagnostics_MC,1319,RulerDomain,11) [Thread 4220] [Port 1] [F12392F48B7E4A55AFDCEE298462574A] Expression 'Reason IN 2 4 8' evaluates to true. 12:05:40:055 (AvDiagnostics_MC,1317,RulerDomain,11) [Thread 4220] [Port 1] [F12392F48B7E4A55AFDCEE298462574A] Rule 'New-Rule' in Domain 'Routing' evaluated to TRUE. 12:05:40:056 (AvDiagnostics_MC,642,RulerDomain,11) [Thread 4220] [Port 1] [F12392F48B7E4A55AFDCEE298462574A] CRulerDomain::FindFirstTrueRule: Rule evaluates to TRUE; Index=[0] 12:05:40:055 (AvDiagnostics_MC,1325,RulerDomain,11) [Thread 4220] [Port 1] [F12392F48B7E4A55AFDCEE298462574A] First Rule to evaluate to TRUE in Domain 'Routing' is 'New-Rule': Action='AD' © 2011 Cisco Systems, Inc. All rights reserved. 46 Diagnostic Example – Basic Call Flow “New-rule” was mistakenly added to Unity causing “Forwarded” calls to subscriber 8005 to go to a “Directory Handler” rather than the default “Attempt Forward”. This will only affect “Forwarded” calls into this subscriber and not a direct call from that subscriber. Rules are evaluated from top to bottom, first “True” match will be executed. © 2011 Cisco Systems, Inc. All rights reserved. 47 Diagnostic Example – Basic Call Flow Prompt Playout 10:59:26:444 (AvDiagnostics_MC,1394,PhraseServer,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] PhraseID - SystemStandard 10:59:26:443 (AvDiagnostics_MC,1394,PhraseServer,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] C:\CommServer\Localize\Prompts\ENU\G711\AvPHGreet\AvPHGreetENU021.wav 10:59:26:444 (AvDiagnostics_MC,1394,PhraseServer,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] C:\CommServer\Localize\Prompts\ENU\G711\AvCommon\Numbers\AvCommonENU208.wav 10:59:26:443 (AvDiagnostics_MC,1394,PhraseServer,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] C:\CommServer\Localize\Prompts\ENU\G711\AvCommon\Numbers\AvCommonENU200.wav 10:59:26:444 (AvDiagnostics_MC,1394,PhraseServer,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] C:\CommServer\Localize\Prompts\ENU\G711\AvCommon\Numbers\AvCommonENU200.wav 10:59:26:443 (AvDiagnostics_MC,1394,PhraseServer,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] C:\CommServer\Localize\Prompts\ENU\G711\AvCommon\Numbers\AvCommonENU052.wav 10:59:26:444 (AvDiagnostics_MC,1394,PhraseServer,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] C:\CommServer\Localize\Prompts\ENU\G711\AvPHGreet\AvPHGreetENU019.wav 10:59:26:443 (AvDiagnostics_MC,1394,PhraseServer,11) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] C:\CommServer\Localize\Prompts\ENU\G711\AvPHGreet\AvPHGreetENU016.wav [10:59:26] Prompt Name - Prompt Name Prompt Name Prompt Name Prompt Name Prompt Name - Prompt Name - prompts.ini file can be viewed in each directory for the text representation of each wav file [AvPHGreetENU021] Text=Extension... [AvCommonENU200] Text=zero (trailing) [AvPHGreetENU019] Text=is not available. [AvCommonENU208] Text=eight (trailing) [AvCommonENU052] Text=five (leading) [AvPHGreetENU016] Text=Record your message at the tone. When you are finished, hang-up or hold for more options. © 2011 Cisco Systems, Inc. All rights reserved. 48 Diagnostic Example – Basic Call Flow Enter / Leaving Functions 10:59:36:600 (**,-1,Conversation,1) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] [Port 1] GetMailBoxStatusbyMailuser: Entering function File: E:\views\TBM17_view\un_Conv1\Scripted\ConvSub\PHGreeting.cpp Line: 1229 10:59:36:724 (AvDiagnostics_MC,1462,MALEx,12) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Mailbox - Logon cn=user 8015 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 on - Thread: 0000107CH Instance: 029531B8H 10:59:36:740 (**,-1,Conversation,12) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] [Port 1] GetMailBoxStatusbyMailuser: spMailUser->get_PrimaryMailbox(spMailbox.put()) returned: 0x00000000 File: E:\views\TBM17_view\un_Conv1\Scripted\ConvSub\PHGreeting.cpp Line: 1239 10:59:36:756 (AvDiagnostics_MC,1440,MALEx,12) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Diagnostic: Mailbox size is 47.00 KB 10:59:36:757 (AvDiagnostics_MC,1440,MALEx,12) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Diagnostic: Mailbox Send Receive Limit is 2411520.00K 10:59:36:756 (AvDiagnostics_MC,1440,MALEx,12) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Diagnostic: Mailbox Send Limit is 2097152.00K 10:59:36:757 (AvDiagnostics_MC,1440,MALEx,12) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Diagnostic: Mailbox Warning Limit is 1991680.00K 10:59:36:757 (**,-1,Conversation,1) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] [Port 1] GetMailBoxStatusbyMailuser: Leaving function hr=0x00000000 File: E:\views\TBM17_view\un_Conv1\Scripted\ConvSub\PHGreeting.cpp Line: 1274 hr=0x00000000 means successful © 2011 Cisco Systems, Inc. All rights reserved. 49 Diagnostic Example – Basic Call Flow 10:59:26:335 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!Root] 10:59:26:366 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!PlayGreeting] 10:59:36:771 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!PlayPostGreetingRecording] 10:59:36:788 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!Java_PreRecord] 10:59:36:787 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!Play_PreRecordMsg] 10:59:36:787 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!Java_RecordMsg] 10:59:36:788 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!RecordMsg] 10:59:44:663 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!Java_RunEditMsg] 10:59:44:662 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!Play_RunEditMsg] 10:59:44:662 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!RunEditMsg] 10:59:44:741 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\messageediting.cde!PlayMessageMenu] 10:59:47:991 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\messageediting.cde!CheckMsgLength] 10:59:47:990 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\messageediting.cde!CheckForSecureMessagingForOutsideCallers] 10:59:47:990 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\messageediting.cde!SendMsg] 10:59:49:334 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\messageediting.cde!ConfirmSend] 10:59:51:005 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!AfterMsg] 10:59:52:177 (AvDiagnostics_MC,1394,CDE,10) [Thread 4220] [Port 1] [8985A5D73D924EFABBE5CEAB14E778BA] Port [1] [c:\commserver\localize\scripts\phgreeting.cde!DoHangup] © 2011 Cisco Systems, Inc. All rights reserved. Next state is Next state is Next state is Next state is Next state is Next state is Next state is Next state is Next state is Next state is Next state is Next state is Next state is Next state is Next state is Next state is Next state is 50 Diagnostic Example – Basic Call Flow © 2011 Cisco Systems, Inc. All rights reserved. 51 Diagnostics Example – MWI • Unity relies on MAPI notifications from Exchange for message count updates • MAPI notification register upon the startup of the AvMsgStoreMonitorSvr Service © 2011 Cisco Systems, Inc. All rights reserved. 52 The process for Generating an MWI Request 1 1. Mailstore sends Notification for - New message - Message read - Message deleted Mailstore NotifyQ Exch or Dom Monitor 2 3 Notifier (caches MWI state) 7 3. Notifier Pops Event from NotifyQ 4 Create 6 Miu Aribiter 5 TSP TAPI PBX/CallManager © 2011 Cisco Systems, Inc. All rights reserved. Resource Manager 2. Monitor Receives Notification -Processes raw notification -Drops Notification into NotifyQ MWI Task 4. MWI Task created if MWI state needs to alter. May retry if needed. 5. Arbiter allocates a port for the MWI dial out 6. MIU ultimately carries out the MWI ON/OFF request(s) 7. NotifyQ receives a SUCCESS or FAILURE notification which the Notifier processes. If SUCCESS, internal MWI state is altered. 53 Diagnostics Example – MWI 09:19:50:609 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6708] Received MonitorMailbox request for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 09:19:50:671 (AvDiagnostics_MC,642,ExchangeMonitor,12) [Thread 6804] (Opening table. pContext->set_InboxTable(lpInboxTbl) for mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85) 09:19:50:672 (AvDiagnostics_MC,1440,MALEx,12) [Thread 6804] Diagnostic: CAvMALExSearchFolderHelpers::GetSearchFolder (FilterNewMsg) 09:19:50:687 (AvDiagnostics_MC,642,ExchangeMonitor,12) [Thread 6804] (Loaded mailbox lpInboxTbl is pointing to:lpInboxTbl>SeekRowApprox(0 1).) 09:19:50:718 (AvDiagnostics_MC,642,ExchangeMonitor,12) [Thread 6804] (InboxTbl->QueryRows( ulMsgCount 0 &pRowSet ) returned with 3 rows in RowSet 1.) 09:19:50:719 (AvDiagnostics_MC,642,ExchangeMonitor,12) [Thread 6804] (AddToMessageCounts:MediaType=2 Priority=1 IsNew=Yes) 09:19:50:718 (AvDiagnostics_MC,642,ExchangeMonitor,12) [Thread 6804] (AddToMessageCounts:MediaType=2 Priority=1 IsNew=Yes) 09:19:50:719 (AvDiagnostics_MC,642,ExchangeMonitor,12) [Thread 6804] (AddToMessageCounts:MediaType=2 Priority=1 IsNew=Yes) 09:19:50:718 (AvDiagnostics_MC,1440,MALEx,12) [Thread 6804] Diagnostic: CAvMALExSearchFolderHelpers::GetSearchFolder (FilterSavedMsgNoEmail) 09:19:50:734 (AvDiagnostics_MC,642,ExchangeMonitor,12) [Thread 6804] (Loaded mailbox lpInboxTbl is pointing to:lpInboxTbl>SeekRowApprox(0 1).) 09:19:50:735 (AvDiagnostics_MC,642,ExchangeMonitor,12) [Thread 6804] (InboxTbl->QueryRows( ulMsgCount 0 &pRowSet ) returned with 0 rows in RowSet 1.) 09:19:50:734 (AvDiagnostics_MC,642,ExchangeMonitor,12) [Thread 6804] (GetMessageCountPackage:Voice=3/0(0/0) Fax=0/0(0/0) Text=0/0(0/0)) 09:19:50:735 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] PostNotifyQPushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_STARTUP_MESSAGE_COUNTS) 09:19:50:734 (AvDiagnostics_MC,642,ExchangeMonitor,13) [Thread 6804] Inbox FID for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 = 0x18000000:0x00000001 09:19:50:735 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] IAvNotifyQ::PushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_STARTUP_MESSAGE_COUNTS) Unity Startup, should see a line for each subscriber that is local to the Unity Server Message counts = {Normal New} / {Normal Read} ( {Urgent New} / {Urgent Read} ) © 2011 Cisco Systems, Inc. All rights reserved. 54 Diagnostics Example – MWI • Callin by subscriber “user 8005”, deletes 3 messages. • MAPI is very CHATTY. 09:25:04:072 (AvDiagnostics_MC,1083,ExchangeMonitor,13) [Thread 6804] [Thread 0x00001A94] Table notification received (TABLE_ROW_MODIFIED for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 (EX2010)) 09:25:04:073 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] PostNotifyQPushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_READ) 09:25:04:072 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] IAvNotifyQ::PushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_READ) 09:25:04:275 (AvDiagnostics_MC,1083,ExchangeMonitor,13) [Thread 6804] [Thread 0x00001A94] Table notification received (TABLE_ROW_MODIFIED for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 (EX2010)) 09:25:04:276 (AvDiagnostics_MC,642,ExchangeMonitor,13) [Thread 6804] CAvMAPIAccessThread::OnNotifyOnThread() TABLE_ROW_MODIFIED: no interesting changes no action required 09:25:04:462 (AvDiagnostics_MC,1083,ExchangeMonitor,13) [Thread 6804] [Thread 0x00001A94] Table notification received (TABLE_ROW_DELETED for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 (EX2010)) 09:25:04:463 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] PostNotifyQPushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_OTHER) 09:25:04:462 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] IAvNotifyQ::PushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_OTHER) 09:25:05:853 (AvDiagnostics_MC,1083,ExchangeMonitor,13) [Thread 6804] [Thread 0x00001A94] Table notification received (TABLE_ROW_MODIFIED for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 (EX2010)) 09:25:05:854 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] PostNotifyQPushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_READ) 09:25:05:853 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] IAvNotifyQ::PushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_READ) 09:25:05:868 (AvDiagnostics_MC,1083,ExchangeMonitor,13) [Thread 6804] [Thread 0x00001A94] Table notification received (TABLE_ROW_MODIFIED for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 (EX2010)) © 2011 Cisco Systems, Inc. All rights reserved. 55 Diagnostics Example – MWI • AvMessageStoreMonitorSvr diag 09:25:05:869 (AvDiagnostics_MC,642,ExchangeMonitor,13) [Thread 6804] CAvMAPIAccessThread::OnNotifyOnThread() TABLE_ROW_MODIFIED: no interesting changes no action required 09:25:05:931 (AvDiagnostics_MC,1083,ExchangeMonitor,13) [Thread 6804] [Thread 0x00001A94] Table notification received (TABLE_ROW_DELETED for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 (EX2010)) 09:25:05:932 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] PostNotifyQPushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_OTHER) 09:25:05:931 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] IAvNotifyQ::PushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_OTHER) 09:25:07:431 (AvDiagnostics_MC,1083,ExchangeMonitor,13) [Thread 6804] [Thread 0x00001A94] Table notification received (TABLE_ROW_MODIFIED for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 (EX2010)) 09:25:07:432 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] PostNotifyQPushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_READ) 09:25:07:431 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] IAvNotifyQ::PushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_READ) 09:25:07:493 (AvDiagnostics_MC,1083,ExchangeMonitor,13) [Thread 6804] [Thread 0x00001A94] Table notification received (TABLE_ROW_MODIFIED for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 (EX2010)) 09:25:07:494 (AvDiagnostics_MC,642,ExchangeMonitor,13) [Thread 6804] CAvMAPIAccessThread::OnNotifyOnThread() TABLE_ROW_MODIFIED: no interesting changes no action required 09:25:07:525 (AvDiagnostics_MC,1083,ExchangeMonitor,13) [Thread 6804] [Thread 0x00001A94] Table notification received (TABLE_ROW_DELETED for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 (EX2010)) 09:25:07:526 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] PostNotifyQPushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_OTHER) 09:25:07:525 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] IAvNotifyQ::PushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_OTHER) © 2011 Cisco Systems, Inc. All rights reserved. 56 Unity Architecture © 2011 Cisco Systems, Inc. All rights reserved. 57 Diagnostics Example – MWI • AvNotifier taking the PushEvent and popping them off the queue • Diags from one of the three messages being deleted 09:25:04:072 (AvDiagnostics_MC,1212,Notifier,21) [Thread 5740] NotifyQ popped eNOTIFYQ_ACTION_MSG_READ [3], mailbox='cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85', arg1=2, arg2=3, arg3=1, varMessageData=<empty> (VT_EMPTY). 09:25:04:073 (AvDiagnostics_MC,642,Notifier,24) [Thread 5740] (DeleteFromMessageCounts:MediaType=Voice(0x2) Priority=Normal Status=New) 09:25:04:072 (AvDiagnostics_MC,642,Notifier,24) [Thread 5740] (AddToMessageCounts:MediaType=Voice(0x2) Priority=Normal Status=Saved) 09:25:04:103 (AvDiagnostics_MC,1225,Notifier,24) [Thread 5740] user 8005 has 2 messages of type Voice(0x2) Urgent and Normal 09:25:04:104 (AvDiagnostics_MC,1225,Notifier,24) [Thread 5740] user 8005 has 0 messages of type Voice(0x2) Urgent Only 09:25:04:103 (AvDiagnostics_MC,1223,Notifier,12) [Thread 5740] user 8005:MWI-1(8005), 2 messages (message just Deleted), current status On, current attempt None © 2011 Cisco Systems, Inc. All rights reserved. 58 Diagnostics Example – MWI • AvNotifierMgr (continued) • Diags from the last messages being deleted and MWI task being executed 09:25:07:525 (AvDiagnostics_MC,1225,Notifier,24) [Thread 5740] user 8005 has 0 messages of type Voice(0x2) Urgent and Normal 09:25:07:526 (AvDiagnostics_MC,1225,Notifier,24) [Thread 5740] user 8005 has 0 messages of type Voice(0x2) Urgent Only 09:25:07:525 (AvDiagnostics_MC,1223,Notifier,12) [Thread 5740] user 8005:MWI-1(8005), 0 messages (message just Deleted), current status On, current attempt Off 09:25:11:571 (AvDiagnostics_MC,1221,Notifier,20) [Thread 6976] Ticket 0x1 granted for resource 2 (requested Any) 09:25:11:572 (AvDiagnostics_MC,1191,Notifier,12) [Thread 4128] MWI Device - MWI Entry AV_MWI_OFF Received: Task user 8005 8005 taskid 1294755907, Port 2 09:25:11:571 (AvDiagnostics_MC,642,Notifier,12) [Thread 4128] (Message Counts NamedProps:Voice=0/0(0/0) Fax=0/0(0/0) Text=0/0(0/0) SendCounts=No) 09:25:11:603 (AvDiagnostics_MC,642,Notifier,12) [Thread 4128] m_pMWIDevice->GetArbiter()->SetMWI(8005 | ) returned [0x00000000; S_OK] 09:25:11:604 (AvDiagnostics_MC,1212,Notifier,21) [Thread 5740] NotifyQ popped eNOTIFYQ_ACTION_MWIOFF_COMPLETE [8], mailbox='cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85', arg1=2, arg2=0, arg3=0, varMessageData='MWI-1' (VT_BSTR). 09:25:11:603 (AvDiagnostics_MC,1250,Notifier,12) [Thread 5740] Decrement Off attempts for 8005 ON=0, OFF=0 09:25:11:618 (AvDiagnostics_MC,1194,Notifier,12) [Thread 4128] Completed MWI task for mailuser=user 8005, extension=8005, switchID=0, status=Off © 2011 Cisco Systems, Inc. All rights reserved. 59 Diagnostics Example – MWI • SvcHost diags • We know: Port 1 = [Device 6], therefore [Device 7]= Port 2 09:25:11:588 **NOT FORMATTED**,AvDiagnostics_MC,162,8172,-1,,SkinnyTSP,18,[Thread 0x00001FEC] [Device 7] CAvTapiLine::SetMwi: szExtension='8005' fMWIon=false pfnCompletionProc=0x4b666a0b dwRequestID=66189 09:25:11:587 **NOT FORMATTED**,AvDiagnostics_MC,162,8172,-1,,SkinnyTSP,13,[Thread 0x00001FEC] [Device 7] Send StationOffHookWithCgpnMessage (28 bytes) callingPartyNumber='8005' (cgpnVoiceMailbox) (lineInstance) 09:25:11:588 **NOT FORMATTED**,AvDiagnostics_MC,162,8172,-1,,SkinnyTSP,21,[Thread 0x00001FEC] [Device 7] CSkinny::GenerateDigits lpszDigits='1000' bUseInterDigitDelay=NoDelay 09:25:11:587 **NOT FORMATTED**,AvDiagnostics_MC,162,8172,-1,,SkinnyTSP,11,[Thread 0x00001FEC] [Device 7] CAvSkinnyCallStatus::RequestStateTransition - RequestedState=<Initiate MWI> (CurrentState=<Idle>) 09:25:11:588 **NOT FORMATTED**,AvDiagnostics_MC,162,8172,-1,,SkinnyTSP,11,[Thread 0x00001FEC] [Device 7] CAvSkinnyCallStatus::AdvanceCurrentState - Old CallState=<Idle> New CallState=<Initiate MWI> 09:25:11:587 **NOT FORMATTED**,AvDiagnostics_MC,162,8172,-1,,SkinnyTSP,11,[Thread 0x00001FEC] [Device 7] CAvSkinnyCallStatus::RequestStateTransition - RequestedState=<Wait For MWI Response> (CurrentState=<Initiate MWI>) 09:25:11:588 **NOT FORMATTED**,AvDiagnostics_MC,162,8172,-1,,SkinnyTSP,11,[Thread 0x00001FEC] [Device 7] CAvSkinnyCallStatus::AdvanceCurrentState - Old CallState=<Initiate MWI> New CallState=<Wait For MWI Response> ... 09:25:11:588 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,12,[Thread 0x00000940] [Device 7] Receive StationCallStateMessage (28 bytes) callState=1=TsOffHook lineInstance=1 callReference=26043803 privacy=0=PrivacyNone precedence=[precedenceLv=4 precedenceDm=0] ... 09:25:11:588 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,12,[Thread 0x00000940] [Device 7] Receive StationStartToneMessage (20 bytes) tone=33=DtInsideDialTone lineInstance=1 callReference=26043803 ... 09:25:11:587 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,12,[Thread 0x00000940] [Device 7] Receive StationDialedNumberMessage (36 bytes) dialedNumber='1000' lineInstance=1 callReference=26043803 “1000” is the MWI off code © 2011 Cisco Systems, Inc. All rights reserved. 60 Diagnostics Example – MWI • SvcHost (Continued) • Since we reach the “TsProceed” state, MWI code must be good. 09:25:11:588 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,12,[Thread 0x00000940] [Device 7] Receive StationCallStateMessage (28 bytes) callState=12=TsProceed lineInstance=1 callReference=26043803 privacy=0=PrivacyNone precedence=[precedenceLv=4 precedenceDm=0] 09:25:11:587 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,12,[Thread 0x00000940] [Device 7] Processing StationCallStateMessage 09:25:11:588 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,11,[Thread 0x00000940] [Device 7] CAvSkinnyCallStatus::AdvanceCurrentState - Old CallState=<Wait For MWI Response> New CallState=<MWI Success> 09:25:11:587 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,24,[Thread 0x00000940] [Device 7] CAvSkinnyCallStatus::SignalCallState: callState=12=TsProceed callReference=26043803 09:25:11:588 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,24,[Thread 0x00000940] [Device 7] CAvSkinnyCallStatus::SignalMWISuccess 09:25:11:587 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,11,[Thread 0x00000940] [Device 7] CAvSkinnyCallStatus::RequestStateTransition - RequestedState=<Wait For MWI Drop> (CurrentState=<MWI Success>) 09:25:11:588 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,11,[Thread 0x00000940] [Device 7] CAvSkinnyCallStatus::AdvanceCurrentState - Old CallState=<MWI Success> New CallState=<Wait For MWI Drop> 09:25:11:587 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,11,[Thread 0x00000940] [Device 7] CAvSkinnyCallStatus::SetTransitionTimeout - dwTimeoutMS=2000 (CurrentState=<Wait For MWI Drop>) 09:25:11:603 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,12,[Thread 0x00000940] [Device 7] Receive StationCallStateMessage (28 bytes) callState=2=TsOnHook lineInstance=1 callReference=26043803 privacy=0=PrivacyNone precedence=[precedenceLv=4 precedenceDm=0] 09:25:11:604 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,12,[Thread 0x00000940] [Device 7] Processing StationCallStateMessage 09:25:11:603 **NOT FORMATTED**,AvDiagnostics_MC,162,2368,-1,,SkinnyTSP,11,[Thread 0x00000940] [Device 7] CAvSkinnyCallStatus::AdvanceCurrentState - Old CallState=<Wait For MWI Drop> New CallState=<Idle> © 2011 Cisco Systems, Inc. All rights reserved. 61 Diagnostics Example – MWI • New Voicemail left for subscriber ‘user 8005’. • AvMsgStoreMonitorSvr Diags. • If No Table notifications, chances are mailbox(s) have gone unmonitored. 09:38:33:040 (AvDiagnostics_MC,1083,ExchangeMonitor,13) [Thread 6804] [Thread 0x00001A94] Table notification received (TABLE_ROW_ADDED for cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 (EX2010)) 09:38:33:041 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] PostNotifyQPushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_NEW) 09:38:33:040 (AvDiagnostics_MC,644,ExchangeMonitor,13) [Thread 6804] IAvNotifyQ::PushEvent (Mailbox: cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85 Action: eNOTIFYQ_ACTION_MSG_NEW) © 2011 Cisco Systems, Inc. All rights reserved. 62 Diagnostics Example – MWI AvNotifierSvr Diags 09:38:33:040 (AvDiagnostics_MC,1212,Notifier,21) [Thread 5740] NotifyQ popped eNOTIFYQ_ACTION_MSG_NEW [2], mailbox='cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85', arg1=2, arg2=3, arg3=1, varMessageData= (VariantType:0x00002011). 09:38:33:041 (AvDiagnostics_MC,642,Notifier,24) [Thread 5740] (AddToMessageCounts:MediaType=Voice(0x2) Priority=Normal Status=New) 09:38:33:056 (AvDiagnostics_MC,1225,Notifier,24) [Thread 5740] user 8005 has 1 messages of type Voice(0x2) Urgent and Normal 09:38:33:057 (AvDiagnostics_MC,1225,Notifier,24) [Thread 5740] user 8005 has 0 messages of type Voice(0x2) Urgent Only 09:38:33:056 (AvDiagnostics_MC,1223,Notifier,12) [Thread 5740] user 8005:MWI-1(8005), 1 messages (message just Added), current status Off, current attempt None 09:38:33:072 (AvDiagnostics_MC,1194,Notifier,12) [Thread 5740] Queued MWI task for mailuser=user 8005, extension=8005, switchID=0, status=On 09:38:33:087 (AvDiagnostics_MC,1205,Notifier,20) [Thread 5740] Ticket 0x10001 request made for resource Any (caps=4 switch=0 cluster=0) for Task user 8005 MWI-1 taskid 2 09:38:33:088 (AvDiagnostics_MC,642,Notifier,12) [Thread 5740] Increment On attempts for 8005 (throttled=false) ON=1 OFF=0 09:38:33:087 (AvDiagnostics_MC,1221,Notifier,20) [Thread 7072] Ticket 0x10001 granted for resource 2 (requested Any) 09:38:33:088 (AvDiagnostics_MC,1191,Notifier,12) [Thread 4128] MWI Device - MWI Entry AV_MWI_ON Received: Task user 8005 8005 taskid 1294756713, Port 2 09:38:33:087 (AvDiagnostics_MC,642,Notifier,12) [Thread 4128] (Message Counts NamedProps:Voice=1/0(0/0) Fax=0/0(0/0) Text=0/0(0/0) SendCounts=No) 09:38:33:119 (AvDiagnostics_MC,642,Notifier,12) [Thread 4128] m_pMWIDevice->GetArbiter()->SetMWI(8005 | ) returned [0x00000000; S_OK] 09:38:33:120 (AvDiagnostics_MC,1212,Notifier,21) [Thread 5740] NotifyQ popped eNOTIFYQ_ACTION_MWION_COMPLETE [7], mailbox='cn=user 8005 cn=Recipients ou=Exchange Administrative Group (FYDIBOHF23SPDLT) o=EFT 85', arg1=2, arg2=0, arg3=0, varMessageData='MWI-1' (VT_BSTR). 09:38:33:119 (AvDiagnostics_MC,1250,Notifier,12) [Thread 5740] Decrement On attempts for 8005 ON=0, OFF=0 09:38:33:134 (AvDiagnostics_MC,1194,Notifier,12) [Thread 4128] Completed MWI task for mailuser=user 8005, extension=8005, switchID=0, status=On “Increment On Attempts… ON=1 OFF=0” : means first attempt to turn on © 2011 Cisco Systems, Inc. All rights reserved. 63 Diagnostics Example – Directory Syncs • DSAD for object owned by the local Unity Server • DsGlobalCatalog for all other objects • Windows Application Event Log Messages Event Type: Event Source: Event Category: Event ID: Date: Time: User: Computer: Description: Information CiscoUnity_DSAD Init 1002 1/11/2011 9:49:08 AM N/A UNITY803 AvDSAD service started. Event Type: Event Source: Event Category: Event ID: Date: Time: User: Computer: Description: Information CiscoUnity_DSAD Stop 1003 1/11/2011 9:47:45 AM N/A UNITY803 AvDSAD service stopped. Event Type: Event Source: Event Category: Event ID: Date: Time: User: Computer: Description: © 2011 Cisco Systems, Inc. All rights reserved. Event Type: Event Source: Event Category: Event ID: Date: Time: User: Computer: Description: Information CiscoUnity_DSGC Init 1003 1/11/2011 9:49:10 AM N/A UNITY803 AvDSGlobalCatalog service started. Information CiscoUnity_DSGC Stop 1004 1/11/2011 9:47:45 AM N/A UNITY803 AvDSGlobalCatalog service stopped. 64 Directory Object Syncs DSAD Local Mail User Local Location Scope DL © 2011 Cisco Systems, Inc. All rights reserved. GlobalCatalog Mail Store Lists Unity Server Objects Remote Location Objects Remote Mail Users Distribution Lists 65 Diagnostics Example – Directory Syncs Sync Intervals (defaults shown from GUSI output) Global Catalog Server = AD.eft.ucbu.local Sync Interval = 15 min(s) Domain Controllers (Domain : DC) eft.ucbu.local : AD.eft.ucbu.local Sync Interval = 2 min(s) Amount of Information allowed per sync USN <= 5000 or 100 MB (prior to Unity 7) USN <= 1000000 or 100 MB (Unity 7+ but works on all) © 2011 Cisco Systems, Inc. All rights reserved. 66 Diagnostics Example – Directory Syncs DC/GC Reconnect Tool (tools depot) initiates full resync Event Type: Information Event Source: CiscoUnity_DSAD Event Category: Info Event ID: 1095 Date: 1/11/2011 Time: 9:54:14 AM User: N/A Computer: UNITY803 Description: The Cisco Unity service that monitors Active Directory (AvDSAD) started a full synchronization cycle with AD.eft.ucbu.local. A message will be written to the event log when this synchronization cycle is complete. Event Type: Information Event Source: CiscoUnity_DSAD Event Category: Info Event ID: 1094 Date: 1/11/2011 Time: 9:54:14 AM User: N/A Computer: UNITY803 Description: The Cisco Unity service that monitors Active Directory (AvDSAD) completed a full synchronization cycle with AD.eft.ucbu.local. © 2011 Cisco Systems, Inc. All rights reserved. 67 Diagnostics Example – Directory Syncs DSAD Diags 09:49:08:606 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Beginning Synchronization. ... 09:49:09:106 (AvDiagnostics_MC,1520,DSAD,17) [Thread 7600] Started synchronizing domain [eft.ucbu.local]. ... 09:49:09:310 (AvDiagnostics_MC,1493,DSAD,15) [Thread 7600] Entering CAvDSADConfig::get_MaxChangeQueueMegaBytes 09:49:09:325 (AvDiagnostics_MC,1495,DSAD,15) [Thread 7600] Exiting CAvDSADConfig::get_MaxChangeQueueMegaBytes [S_OK: The operation completed successfully. ] 09:49:09:326 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] DirChangeQueue size is 0MB MaxChangeQueueMB is 100MB. ... 09:49:09:403 (AvDiagnostics_MC,1493,DSAD,12) [Thread 7600] Entering CAvADConnection::Connect 09:49:09:404 (AvDiagnostics_MC,1503,DSAD,12) [Thread 7600] Connecting to server: AD.eft.ucbu.local. ... 09:49:09:404 (AvDiagnostics_MC,1495,DSAD,12) [Thread 7600] Exiting CAvADConnection::Connect [S_OK: The operation completed successfully. ] © 2011 Cisco Systems, Inc. All rights reserved. 68 Diagnostics Example – Directory Syncs • DSAD Daigs (continued) • Current Highest USN - Previous Highest USN = 9 changes • Not all AD changes deal with Unity 09:49:09:435 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] 9 directory changes occurred since previous synch. Querying against the next 9 changes. 09:49:09:434 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Previous highest USN: 56276 Current highest USN: 56285 MaxUSNIncrement: 1000000. ... 09:49:09:450 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Changed MailUser LDAP filter: (&(ciscoEcsbuObjectType=1)(|(ciscoEcsbuUMLocationObjectId=\01\00\07\00\30\00\39\00\3A\00\7B\00\39\00\44\00\33\00\42\00\32\00\41\00\38\0 0\45\00\2D\00\38\00\38\00*))(uSNChanged>=56277)). 09:49:09:451 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Deleted MailUser LDAP filter: (&(ciscoEcsbuObjectType=1)(isDeleted=TRUE)(uSNChanged>=56277)). ... 09:49:09:451 (AvDiagnostics_MC,1495,DSAD,1) [Thread 7600] Exiting CAvADSearch::Next [S_ADS_NOMORE_ROWS: <unknown>] 09:49:09:450 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Enqueued 0 change requests of type AV_DIRCHANGE_MODIFY for AVOBJECTTYPE_MAILUSER objects. ... 09:49:09:467 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Enqueued 0 change requests of type AV_DIRCHANGE_DELETE for AVOBJECTTYPE_MAILUSER objects. © 2011 Cisco Systems, Inc. All rights reserved. 69 Diagnostics Example – Directory Syncs DSAD Diags (continued) 09:49:09:482 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Changed Location LDAP filter: (&(ciscoEcsbuObjectType=9)(|(ciscoEcsbuUMLocationObjectId=\01\00\07\00\30\00\39\00\3A\00\7B\00\39\00\44\00\33\00\42\00\32\00\41\00\38\0 0\45\00\2D\00\38\00\38\00*))(uSNChanged>=56277)). 09:49:09:481 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Deleted Location LDAP filter: (&(ciscoEcsbuObjectType=9)(isDeleted=TRUE)(uSNChanged>=56277)). ... 09:49:09:497 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Enqueued 0 change requests of type AV_DIRCHANGE_MODIFY for AVOBJECTTYPE_LOCATION objects. ... 09:49:09:498 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Enqueued 0 change requests of type AV_DIRCHANGE_DELETE for AVOBJECTTYPE_LOCATION objects. ... 09:49:09:528 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Changed ScopeDL LDAP filter: (&(objectCategory=group)(uSNChanged>=56277)). 09:49:09:529 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Deleted ScopeDL LDAP filter: (&(isDeleted=TRUE)(uSNChanged>=56277)(|(!(ciscoEcsbuUMLocationObjectId=*))(ciscoEcsbuUMLocationObjectId=2))). ... 09:49:09:575 (AvDiagnostics_MC,1503,DSAD,17) [Thread 7600] Finished synchronization. ScopeDL : Scope Distribution List © 2011 Cisco Systems, Inc. All rights reserved. 70 Diagnostics Example – Directory Syncs DsGlobalCatalog Diags 09:49:10:200 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Beginning Synchronization. ... 09:49:10:606 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] 12 directory changes occurred since previous synch. Querying against the next 12 changes. 09:49:10:607 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Previous highest USN: 56274 Current highest USN: 56286 MaxUSNIncrement: 1000000. ... 09:49:10:794 (AvDiagnostics_MC,1689,DSGlobalCatalog,15) [Thread 1216] Method CAvDSGCConfig::get_DefaultMailboxStoreFilter exited [0x00000000: S_OK]. 09:49:10:795 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Changed Mailboxstore LDAP filter: (&(objectCategory=CN=ms-ExchPrivate-MDB CN=Schema CN=Configuration DC=eft DC=ucbu DC=local)(uSNChanged>=56275)). 09:49:10:794 (AvDiagnostics_MC,1702,DSGlobalCatalog,10) [Thread 1216] Synchronizing changed AVOBJECTTYPE_MAILBOX_STORE objects. LDAP Filter: (&(objectCategory=CN=ms-Exch-Private-MDB CN=Schema CN=Configuration DC=eft DC=ucbu DC=local)(uSNChanged>=56275)). ... 09:49:10:809 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Enqueued 0 change requests of type AV_DIRCHANGE_MODIFY for AVOBJECTTYPE_MAILBOX_STORE objects. ... 09:49:10:841 (AvDiagnostics_MC,1702,DSGlobalCatalog,10) [Thread 1216] Synchronizing deleted AVOBJECTTYPE_MAILBOX_STORE objects. LDAP Filter: (&(uSNChanged>=56275)(&(objectclass=msExchPrivateMDB)(isdeleted=TRUE))). ... 09:49:10:842 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Enqueued 0 change requests of type AV_DIRCHANGE_DELETE for AVOBJECTTYPE_MAILBOX_STORE objects. © 2011 Cisco Systems, Inc. All rights reserved. 71 Diagnostics Example – Directory Syncs DSGlobalCatallog (continued) 09:49:10:889 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Changed Server LDAP filter: (&(ciscoEcsbuObjectType=14)(ciscoEcsbuUMLocationObjectId=*)(uSNChanged>=56275)). 09:49:10:889 (AvDiagnostics_MC,1702,DSGlobalCatalog,10) [Thread 1216] Synchronizing changed AVOBJECTTYPE_SERVER objects. LDAP Filter: (&(ciscoEcsbuObjectType=14)(ciscoEcsbuUMLocationObjectId=*)(uSNChanged>=56275)). ... 09:49:10:903 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Enqueued 0 change requests of type AV_DIRCHANGE_MODIFY for AVOBJECTTYPE_SERVER objects. ... 09:49:10:935 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Deleted Server LDAP filter: (&(ciscoEcsbuObjectType=14)(uSNChanged>=56275)(isDeleted=TRUE)). 09:49:10:935 (AvDiagnostics_MC,1702,DSGlobalCatalog,10) [Thread 1216] Synchronizing deleted AVOBJECTTYPE_SERVER objects. LDAP Filter: (&(ciscoEcsbuObjectType=14)(uSNChanged>=56275)(isDeleted=TRUE)). ... 09:49:10:950 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Enqueued 0 change requests of type AV_DIRCHANGE_DELETE for AVOBJECTTYPE_SERVER objects. ... 09:49:10:998 (AvDiagnostics_MC,1694,DSGlobalCatalog,15) [Thread 1216] DefaultLocationIdForLDAPQuery value: [\01\00\07\00\30\00\39\00\3A\00\7B\00\39\00\44\00\33\00\42\00\32\00\41\00\38\00\45\00\2D\00\38\00\38\00*]. 09:49:10:998 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Changed Location LDAP filter: (&(ciscoEcsbuObjectType=9)(ciscoEcsbuUMLocationObjectId=*)(!(ciscoEcsbuUMLocationObjectId=\01\00\07\00\30\00\39\00\3A\00\7B\00\39\00\ 44\00\33\00\42\00\32\00\41\00\38\00\45\00\2D\00\38\00\38\00*))(uSNChanged>=56275)). 09:49:10:997 (AvDiagnostics_MC,1702,DSGlobalCatalog,10) [Thread 1216] Synchronizing changed AVOBJECTTYPE_LOCATION objects. LDAP Filter: (&(ciscoEcsbuObjectType=9)(ciscoEcsbuUMLocationObjectId=*)(!(ciscoEcsbuUMLocationObjectId=\01\00\07\00\30\00\39\00\3A\00\7B\00\39\00\ 44\00\33\00\42\00\32\00\41\00\38\00\45\00\2D\00\38\00\38\00*))(uSNChanged>=56275)). ... 09:49:11:013 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Enqueued 0 change requests of type AV_DIRCHANGE_MODIFY for AVOBJECTTYPE_LOCATION objects. © 2011 Cisco Systems, Inc. All rights reserved. 72 Diagnostics Example – Directory Syncs DSGlobalCatalog (continued) 09:49:11:045 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Deleted Location LDAP filter: (&(ciscoEcsbuObjectType=9)(isDeleted=TRUE)(uSNChanged>=56275)). 09:49:11:044 (AvDiagnostics_MC,1702,DSGlobalCatalog,10) [Thread 1216] Synchronizing deleted AVOBJECTTYPE_LOCATION objects. LDAP Filter: (&(ciscoEcsbuObjectType=9)(isDeleted=TRUE)(uSNChanged>=56275)). ... 09:49:11:045 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Enqueued 0 change requests of type AV_DIRCHANGE_DELETE for AVOBJECTTYPE_LOCATION objects. ... 09:49:11:107 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Changed MailUser LDAP filter: (&(ciscoEcsbuObjectType=1)(ciscoEcsbuUMLocationObjectId=*)(!(ciscoEcsbuUMLocationObjectId=\01\00\07\00\30\00\39\00\3A\00\7B\00\39\00\4 4\00\33\00\42\00\32\00\41\00\38\00\45\00\2D\00\38\00\38\00*))(uSNChanged>=56275)). 09:49:11:106 (AvDiagnostics_MC,1702,DSGlobalCatalog,10) [Thread 1216] Synchronizing changed AVOBJECTTYPE_MAILUSER objects. LDAP Filter: (&(ciscoEcsbuObjectType=1)(ciscoEcsbuUMLocationObjectId=*)(!(ciscoEcsbuUMLocationObjectId=\01\00\07\00\30\00\39\00\3A\00\7B\00\39\00\4 4\00\33\00\42\00\32\00\41\00\38\00\45\00\2D\00\38\00\38\00*))(uSNChanged>=56275)). ... 09:49:11:107 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Enqueued 0 change requests of type AV_DIRCHANGE_MODIFY for AVOBJECTTYPE_MAILUSER objects. ... 09:49:11:139 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Deleted MailUser LDAP filter: (&(ciscoEcsbuObjectType=1)(uSNChanged>=56275)(|(!(ciscoEcsbuUMLocationObjectId=*))(isDeleted=TRUE))). 09:49:11:138 (AvDiagnostics_MC,1702,DSGlobalCatalog,10) [Thread 1216] Synchronizing deleted AVOBJECTTYPE_MAILUSER objects. LDAP Filter: (&(ciscoEcsbuObjectType=1)(uSNChanged>=56275)(|(!(ciscoEcsbuUMLocationObjectId=*))(isDeleted=TRUE))). ... 09:49:11:153 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Enqueued 0 change requests of type AV_DIRCHANGE_DELETE for AVOBJECTTYPE_MAILUSER objects. © 2011 Cisco Systems, Inc. All rights reserved. 73 Diagnostics Example – Directory Syncs DSGlobalCatalog Diags (continued) 09:49:11:184 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Changed DL LDAP filter: (&(ciscoEcsbuObjectType=2)(ciscoEcsbuUMLocationObjectId=*)(uSNChanged>=56275)). 09:49:11:185 (AvDiagnostics_MC,1702,DSGlobalCatalog,10) [Thread 1216] Synchronizing changed AVOBJECTTYPE_DISTLIST objects. LDAP Filter: (&(ciscoEcsbuObjectType=2)(ciscoEcsbuUMLocationObjectId=*)(uSNChanged>=56275)). ... 09:49:11:420 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Enqueued 1 change requests of type AV_DIRCHANGE_MODIFY for AVOBJECTTYPE_DISTLIST objects. ... 09:49:11:467 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Deleted DL LDAP filter: (&(ciscoEcsbuObjectType=2)(uSNChanged>=56275)(|(!(ciscoEcsbuUMLocationObjectId=*))(isDeleted=TRUE))). 09:49:11:466 (AvDiagnostics_MC,1702,DSGlobalCatalog,10) [Thread 1216] Synchronizing deleted AVOBJECTTYPE_DISTLIST objects. LDAP Filter: (&(ciscoEcsbuObjectType=2)(uSNChanged>=56275)(|(!(ciscoEcsbuUMLocationObjectId=*))(isDeleted=TRUE))). ... 09:49:11:528 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Enqueued 0 change requests of type AV_DIRCHANGE_DELETE for AVOBJECTTYPE_DISTLIST objects. ... 09:49:11:592 (AvDiagnostics_MC,1702,DSGlobalCatalog,17) [Thread 1216] Finished Synchronization. 09:49:11:591 (AvDiagnostics_MC,1717,DSGlobalCatalog,17) [Thread 1216] Total synchronization cycle lasted [0] hour, [0] minutes and [1] seconds. © 2011 Cisco Systems, Inc. All rights reserved. 74 Diagnostics Example – Directory Syncs • To see what objects will be acted on, cut/paste LDAP line from DSAD/DSGlobalt Catalog diags into ADSIEDIT. • ADSIEDIT can be found in the under the “techtools” directory where unity was installed \commserver. © 2011 Cisco Systems, Inc. All rights reserved. 75 Diagnostics Example – Directory Syncs LDAP results show the objects found is the “MyGroup” group object © 2011 Cisco Systems, Inc. All rights reserved. 76 Diagnostics Example – Directory Syncs • Micro DirChangeWriter 10+ enabled • User ‘6600 user’ added to the mygroup Public Distribution List via “Active Directory Users and Computers” 11:09:36:515 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] CHANGEREQ: [LogId] 58 [ReqType] AV_DIRCHANGE_CREATE | AV_DIRCHANGE_GLOBAL_CATALOG [ObjType] AVOBJECTTYPE_RECIPIENT [DirLogId] 62 [LocLogId] 63 [PropIdsId] 64 [PropValsId] 65 11:09:36:514 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] VARIANT: [LogId] 62 [vt] VT_BSTR [Data] e7112216f1bd174886989d0c20d86f77 11:09:36:515 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] OBJID: [LogId] 63 [Hdr] 0x00070001 [ObjType] AVOBJECTTYPE_LOCATION [ID] {9D3B2A8E-88EC-4ACF-BD05-EEF7E1DCE24A} 11:09:36:514 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] IDLIST: [LogId] 64 [NumElems] 6 11:09:36:515 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] IDLIST: [LogId] 64 [PropId] AVP_ALIAS [ExpType] VT_BSTR 11:09:36:514 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] IDLIST: [LogId] 64 [PropId] AVP_DISPLAY_NAME [ExpType] VT_BSTR 11:09:36:515 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] IDLIST: [LogId] 64 [PropId] AVP_DIRECTORY_ID [ExpType] VT_BSTR 11:09:36:514 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] IDLIST: [LogId] 64 [PropId] AVP_OBJECT_TYPE [ExpType] VT_I4 11:09:36:515 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] IDLIST: [LogId] 64 [PropId] AVP_PARENT_DIRECTORY_ID [ExpType] VT_BSTR 11:09:36:514 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] IDLIST: [LogId] 64 [PropId] AVP_PARENT_ALIAS [ExpType] VT_BSTR 11:09:36:515 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] VARIANT: [LogId] 65 [vt] VT_ARRAY | VT_VARIANT [Dim] 1 [ElemSize] 16 [NumElems] 6 [LBound] 0 [Data(first 32 bytes)] 08 00 00 00 00 00 00 00 84 56 1E 00 00 00 00 00 08 00 00 00 00 00 00 00 8C 13 18 00 00 00 00 00 11:09:36:514 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] VARIANT: [LogId] 65 [vt] VT_BSTR [Data] 6600user 11:09:36:515 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] VARIANT: [LogId] 65 [vt] VT_BSTR [Data] 6600 user … 11:09:36:515 (AvDiagnostics_MC,1889,Directory Change Writer,16) [Thread 2028] VARIANT: [LogId] 65 [vt] VT_BSTR [Data] mygroup © 2011 Cisco Systems, Inc. All rights reserved. 77 Diagnostics Example – Directory Syncs • DSAD diags • Used SAWeb to remove ‘user 8005’ from the mygroup PDL. • SA used the DSAD service to make the request direct to AD rather than waiting for a normal AD sync. 11:20:33:377 (AvDiagnostics_MC,1573,DSAD,20) [Thread 3536] Object: [CN=user 8005 CN=Users DC=eft DC=ucbu DC=local] Property: [IADs::get_ADsPath] Value: [LDAP://AD.eft.ucbu.local:389/<GUID=5fd7a401072de049a93a048c259d7a2f>] AccessType: [Read] DC: [LD]. 11:20:33:376 (AvDiagnostics_MC,1495,DSAD,12) [Thread 3536] Exiting CAvADConnection::GetADsPathFromGUID [S_OK: The operation completed successfully. ] 11:20:33:377 (AvDiagnostics_MC,1493,DSAD,12) [Thread 3536] Entering CAvADConnection::RemoveMember ... 11:20:33:376 (AvDiagnostics_MC,1495,DSAD,12) [Thread 3536] Exiting CAvADConnection::RemoveMember [S_OK: The operation completed successfully. ] © 2011 Cisco Systems, Inc. All rights reserved. 78 Agenda • The playing field. • Setting the traces. • Customizing the traces. • Gathering everything up. • Diagnotistic Examples • Q&A. © 2011 Cisco Systems, Inc. All rights reserved. 79 Q&A © 2011 Cisco Systems, Inc. All rights reserved. 80 © 2011 Cisco Systems, Inc. All rights reserved. 81 Miscellaneous Information © 2011 Cisco Systems, Inc. All rights reserved. 82 Call States TAPI calls are handled in ‘states’: LINECALLSTATE_IDLE LINECALLSTATE_OFFERING LINECALLSTATE_CONNECTED Unity CCM MIU (CME) Etc. AvWave TAPI SCCP (Skinny) calls are handled in ‘states’: TsOnHook TsRingIn TsConnect Etc. © 2011 Cisco Systems, Inc. All rights reserved. Call Control (SCCP) WAV TSP Au di oS (RT tream P) Call Control (???) IP Phone or Gateway 83 Call States TSP has internal states built around the Skinny and TAPI call states • Responses to incoming Skinny or TAPI events depend on the current state of the TSP • Unexpected behavior, or even a locked port, may occur if states become out of sync between TAPI, TSP and Skinny • For full design see FDS: EDCS-396019 • Updated state diagrams: EDCS-494253 © 2011 Cisco Systems, Inc. All rights reserved. 84 Basic Call States REGISTER COMPLETED INITIATE MAKECALL SEND OffHook WAIT FOR MAKECALL OFFHOOK RECEIVE CallState (OffHook) AND StartTone (DialTone) MAKECALL OFFHOOK (Drop Requested) OR RECEIVE StartTone (Busy/ Reorder) OR TIMEOUT (MakeCall Requested) IDLE WAIT FOR MAKECALL RINGING SEND KeyPadButton RECEIVE CallState (RingIn) WAIT FOR ANSWER REQUESTED (Answer Requested) ANSWER REQUESTED RECEIVE CallState (OnHook) INITIATE MWI (MWI) WAIT FOR CONNECT (Drop Requested) OR RECEIVE StartTone (Busy/ Reorder) OR TIMEOUT (Drop Requested) OR RECEIVE StartTone (Busy/ Reorder) INITIATE DROP (Drop Requested) CONNECTED INITIATE SUPER XFER (S Xfer) WAIT FOR ONHOOK RECEIVE CallState (OnHook) RECEIVE (CallState(Connected) AND StartMediaTransmission) OR StartTone(OnHold) (Supervised Transfer Requested) SEND OnHook (Unity Closes Call) RECEIVE CallState (Proceed) OR CallState (RingOut) SEND OffHook (Reject Requested) (Reject Requested) RECEIVE CallState (OnHook) RECEIVE CallState (OnHook) (Blind Transfer Requested) INITIATE BLIND XFER (B Xfer) RECEIVE CallState (OnHook) RECEIVE CallState (OnHook) PORT LOCKED (Unity Closes Call) TIMEOUT (Unity Closes Call) © 2011 Cisco Systems, Inc. All rights reserved. NEW CALL PENDING CLOSE CALL RECEIVE CallState (RingIn) WAIT FOR CLOSE CALL (Unity Closes Call) 85 MWI Calls (CCM & CME) IDLE (Basic Call) (MWI Requested) CCM INITIATE MWI IDLE (Basic Call) SEND OffHookWithCpgn AND KeyPadButton CME (MWI Requested) INITIATE MWI SEND OffHookWithCpgn RECEIVE CallState (OnHook) WAIT FOR MWI DIALTONE RECEIVE CallState (OffHook) RECEIVE CallState (OnHook) MWI DIALTONE SEND KeyPadButton RECEIVE CallState (OnHook) RECEIVE CallState (OnHook) WAIT FOR MWI RESPONSE RECEIVE CallState (Proceed) MWI SUCCESS RECEIVE StartTone (Reorder) OR OpenReceiveChannel OR TIMEOUT RECEIVE CallState (OnHook) RECEIVE CallState (OnHook) (Timeout) RECEIVE CallState (OnHook) WAIT FOR MWI DROP WAIT FOR MWI RESPONSE MWI SUCCESS RECEIVE StartTone (Alerting) RECEIVE StartTone (Reorder) OR TIMEOUT (Timeout) SEND OnHook © 2011 Cisco Systems, Inc. All rights reserved. MWI FAILURE RECEIVE CallState (OnHook) WAIT FOR MWI DROP SEND OnHook MWI FAILURE 86 Blind Transfer (CCM) WAIT FOR BLIND XFER OFFHOOK CONNECTED (Basic Call) RECEIVE CallState (OffHook/Call2) AND StartTone (DialTone) (Blind Transfer Requested) INITIATE BLIND XFER RECEIVE CallState (OnHook/Call1) SEND SoftKey (Transfer) RECEIVE CallState (OnHook/Call1) BLIND XFER OFFHOOK BLIND XFER RINGING RECEIVE CallState (RingOut/Call2) OR CallState (Connected/ Call2) RECEIVE CallState (Proceed/Call2) SEND SoftKey (Transfer) RECEIVE CallState (RingOut/Call2) OR CallState (Connected/ Call2) OR Timeout (500ms) WAIT FOR BLIND XFER ONHOOK (drop requested) WAIT FOR BLIND XFER RINGING RECEIVE StartTone (Reorder/Busy) RECEIVE CallState (OnHook/Call1) RECEIVE CallState (Connected/Call1) AND StartMedia Transmission SEND SoftKey (Resume) SEND KeyPadButton RECEIVE CallState (RingOut/Call2) OR CallState (Connected/ Call2) WAIT FOR BLIND XFER RECALLED INITIATE BLIND XFER RECALL CONNECTED (Basic Call) RECEIVE CallState (OnHook/Call2) RECEIVE CallState (OnHook/Call1) RECEIVE StartTone (Reorder/Busy) OR OpenReceiveChannel OR (fwd loop detected) WAIT FOR BLIND XFER BUSY WAIT FOR BLIND XFER REJECTED INITIATE DROP (Basic Call) SEND SoftKey (EndCall) RECEIVE CallState (OnHook/Call1) INITIATE BLIND XFER REJECT IDLE (Basic Call) RECEIVE StartTone (Reorder/Busy) OR (fwd loop detected) (Unity Closes Call) (Unity Closes Call) INITIATE BLIND XFER ABANDON Consult call does exist SEND SoftKey (EndCall) WAIT FOR BLIND XFER ABANDONED RECEIVE CallState (OnHook/ Call2) WAIT FOR CLOSE CALL (Basic Call) Consult call does NOT exist © 2011 Cisco Systems, Inc. All rights reserved. 87 Blind Transfer (CME) WAIT FOR BLIND XFER OFFHOOK CONNECTED (Basic Call) RECEIVE CallState (OnHook/Call1) RECEIVE SetLamp (Transfer/Flash) AND StartTone (DialTone) (Blind Transfer Requested) INITIATE BLIND XFER WAIT FOR BLIND XFER RECALLED BLIND XFER OFFHOOK SEND SoftKey (Transfer) RECEIVE CallState (Connected/Call1) AND StartMedia Transmission SEND SoftKey (Resume) INITIATE BLIND XFER RECALL RECEIVE CallState (OnHook/Call1) CONNECTED (Basic Call) SEND KeyPadButton BLIND XFER RINGING RECEIVE CallState (RingOut/Call2) OR CallState (Connected/ Call2) RECEIVE StartTone (Alerting) OR CallState (Connected/ Call2) WAIT FOR BLIND XFER RINGING RECEIVE CallState (OnHook/Call1) INITIATE DROP (Basic Call) RECEIVE StartTone (Reorder/Busy) OR (fwd loop detected) SEND SoftKey (Transfer) WAIT FOR BLIND XFER ONHOOK IDLE (Basic Call) (drop requested) RECEIVE CallState (OnHook/Call1) AND SetLamp (Line/Off) WAIT FOR CLOSE CALL (Basic Call) (Unity Closes Call) RECEIVE CallState (OnHook/ Call2) WAIT FOR BLIND XFER ABANDONED SEND SoftKey (EndCall) Consult call does exist INITIATE BLIND XFER ABANDON Consult call does NOT exist © 2011 Cisco Systems, Inc. All rights reserved. 88 Supervised Transfer (CCM) RECEIVE CallState (OnHook/Call1) CONNECTED (Basic Call) WAIT FOR SUPER XFER COMPLETION INITIATE SUPER XFER RECALL (Supervised Transfer Requested) SEND SoftKey (Transfer) SEND SoftKey (Resume) INITIATE SUPER XFER SEND SoftKey (Transfer) WAIT FOR CONSULT CALL GENERATION (Supervised Transfer Completion Requested) RECEIVE CallState (OnHook/ Call1) RECEIVE CallState (OffHook/Call2) AND StartTone (DialTone) CONSULT CALL GENERATED RECEIVE CallState (OnHook/ Call2) INITIATE SUPER XFER COMPLETION RECEIVE CallState (OnHook/Call1) RECEIVE CallState (OnHook/ Call2) CONSULT CALL CONNECTED WAIT FOR SUPER XFER RECALLED INITIATE SUPER XFER ABANDON RECEIVE CallState (OnHook/ Call1) IF CONSULT CALL DOES NOT EXIST RECEIVE CallState (OnHook/ Call1) SEND SoftKey (EndCall) WAIT FOR SUPER XFER ABANDONED RECEIVE CallState (Connected/Call1) AND StartMediaTransmission RECEIVE CallState (OnHook/Call2) CONNECTED (Basic Call) WAIT FOR CLOSE CALL (Basic Call) (Recall Requested) OR (fwd loop detected) (Unity Closes Call) (Unity Closes Call) RECEIVE CallState (Connected/Call2) AND StartMediaTransmission RECEIVE StartTone (Reorder/Busy) OR TIMEOUT (RNA) OR (fwd loop detected) WAIT FOR CONSULT CALL CONNECTED RECEIVE CallState (OnHook/ Call1) RECEIVE CallState (OnHook/ Call2) RECEIVE CallState (Proceed/Call2) OR CallState (RingOut/Call2) SEND KeyPadButton RECEIVE CallState (OnHook /Call1) © 2011 Cisco Systems, Inc. All rights reserved. WAIT FOR CONSULT CALL RINGING RECEIVE CallState (OnHook/ Call2) RECEIVE StartTone (Reorder/Busy) OR OpenReceiveChannel OR TIMEOUT (RNA) WAIT FOR CONSULT CALL DROPPED RECEIVE CallState (OnHook/ Call1) IDLE (Basic Call) SEND SoftKey (EndCall) INITIATE CONSULT CALL DROP RECEIVE CallState (OnHook/ Call1) 89 Supervised Transfer (CME) RECEIVE CallState (OnHook/Call1) AND CallState (OnHook/Call1) CONNECTED (Basic Call) WAIT FOR SUPER XFER COMPLETION INITIATE SUPER XFER RECALL (Supervised Transfer Requested) SEND SoftKey (Transfer) SEND SoftKey (Resume) INITIATE SUPER XFER INITIATE SUPER XFER COMPLETION WAIT FOR SUPER XFER RECALLED SEND SoftKey (Transfer) (Supervised Transfer Completion Requested) RECEIVE CallState (Connected/Call1) AND StartMediaTransmission RECEIVE CallState (OnHook/Call2) CONNECTED (Basic Call) WAIT FOR CLOSE CALL (Basic Call) WAIT FOR CONSULT CALL GENERATION RECEIVE CallState (OnHook/ Call1) RECEIVE SetLamp (Transfer/Flash) AND StartTone (DialTone) CONSULT CALL GENERATED RECEIVE CallState (OnHook/ Call2) CONSULT CALL CONNECTED INITIATE SUPER XFER ABANDON RECEIVE CallState (OnHook/ Call1) IF CONSULT CALL DOES NOT EXIST RECEIVE CallState (OnHook/ Call1) (Recall Requested) OR (fwd loop detected) (Unity Closes Call) SEND SoftKey (EndCall) WAIT FOR SUPER XFER ABANDONED (Unity Closes Call) RECEIVE CallState (Connected/Call2) AND StartMediaTransmission RECEIVE CallState (OnHook/ Call1) SEND KeyPadButton WAIT FOR CONSULT CALL CONNECTED TIMEOUT OR (fwd loop detected) RECEIVE CallState (OnHook/Call2) OR StartTone (Reorder/Busy) RECEIVE CallState (RingOut/Call2) RECEIVE CallState (OnHook /Call1) © 2011 Cisco Systems, Inc. All rights reserved. WAIT FOR CONSULT CALL RINGING RECEIVE CallState (OnHook/Call2) OR StartTone (Reorder/Busy) TIMEOUT RECEIVE CallState (OnHook/ Call2) WAIT FOR CONSULT CALL DROPPED RECEIVE CallState (OnHook/ Call1) IDLE (Basic Call) SEND SoftKey (EndCall) INITIATE CONSULT CALL DROP RECEIVE CallState (OnHook/ Call1) 90