Date:
Document:
Application Note for
Witness Systems
Product:
Witness Systems
IP Recording Suite
Call Manager
V 3.3 CallManager
Revision:
2
Author:
Maxine Dye
October 15, 2003
Application Note for Witness Systems
Contact Store IP
Table of Contents
1
2
3
4
5
General Information .................................................................................................... 3
Main Contact Information........................................................................................... 3
Witness Contacts......................................................................................................... 3
Witness Support Contact Information ........................................................................ 4
Witness Support Detail ............................................................................................... 4
5.1
General Information ............................................................................................ 4
5.2
Severity Description and Response Times ......................................................... 5
5.3
Support Level Definitions ................................................................................... 5
5.4
Call Flow Description ......................................................................................... 6
6 Witness Support Escalation Procedures...................................................................... 7
7 Witness Product Information ...................................................................................... 8
8 Witness Product Overview ......................................................................................... 8
9 Witness Product Features ............................................................................................ 8
10
Witness Product Performance (as tested) ............................................................... 8
11
Witness Product Security Considerations ............................................................. 10
12
Packaging, Bundling, Upgrading .......................................................................... 10
13
Overview of Integrated Solution (as tested) ......................................................... 10
14
Network Designs/Topologies (as tested) .............................................................. 11
14.1 Sample network designs/topologies .................................................................. 11
14.1.1
Simple Topologies .................................................................................... 11
14.1.2
Single Site, Single Gateway and E1/T1 Trunk ......................................... 12
14.1.3
Single Site, Dual E1/T1 Trunks ................................................................ 12
14.1.4
Hub and Satellite Sites .............................................................................. 13
14.1.5
More Complex Scenarios .......................................................................... 14
15
Witness Product Requirements ............................................................................. 15
16
Witness Product Installation Procedures (as tested) ............................................. 16
17
Witness Product Isolation Procedures .................................................................. 16
18
General Witness Product Considerations.............................................................. 16
18.1
Limitations of this release ................................................................................. 16
18.2 If you alter your server ...................................................................................... 17
18.3 If you alter your network .................................................................................. 17
18.4 If you alter your IP telephony system ............................................................... 18
19
Troubleshooting .................................................................................................... 19
19.1 Debugging and Diagnostics .............................................................................. 19
19.2 Error Messages.................................................................................................. 20
19.3 Tools / Aids ....................................................................................................... 22
19.4 List of Commands ............................................................................................. 24
20
Redundancy and Recovery ................................................................................... 25
20.1 Redundancy....................................................................................................... 25
20.2 Recovery ........................................................................................................... 25
20.3 Dual Spanning ................................................................................................... 25
21
Interoperability Test Results ................................................................................. 25
22
Cisco Product Information (as tested)................................................................... 25
23
Cisco Product Interfaces (as tested) ...................................................................... 26
24
Cisco Product Requirements (as tested) ............................................................... 26
25
CallManager Device Table ................................................................................... 26
Appendix A ....................................................................................................................... 27
Glossary of Terms:
API
Application Program Interface
A set of routines used by an application program to direct the
performance of procedures by the computer's operating system.
AVVID
Architecture for Voice, Video and Integrated Data.
CRM
Customer Relationship Management
CTI
The practice of using a computer to control one or more telephone
and communications functions.
IVR
Interactive Voice Response
A computer that operates through the telephone system, in which
input commands and data are transmitted to the computer as
spoken words and numbers, or tones and dial pulses generated by a
telephone instrument; and output instructions and data are received
from the computer as prerecorded or synthesized speech.
IIS
Internet Information Server
Microsoft's brand of Web server software, utilizing HTTP
(Hypertext Transfer Protocol) to deliver World Wide Web
documents. It incorporates various functions for security, allows
for CGI (Common Gateway Interfaces) programs, and also
provides for Gopher and FTP (File Transfer Protocol) services.
IP
Internet Protocol
LAN
Local Area Network
PSTN
Public Switched Telephone Network
Public telephone system.
RAID
Redundant Array of Independent Disks
A data storage method in which data, along with information used for
error correction, such as parity bits or Hamming codes, is distributed
among two or more hard disk drives in order to improve performance
and reliability. The hard disk array is governed by array management
software and a disk controller, which handles the error correction.
RTP
Real-Time Protocol
An Internet-standard network transport protocol used in delivering realtime data, including audio and video.
SCCP
Skinny Client Control Protocol
A Cisco protocol used for communication between a gateway and
Cisco CallManager.
SPAN
Switch Port Analyser.
1
VLAN
Virtual Local Area Network
A local area network consisting of groups of hosts that are on physically
different segments but that communicate as though they were on the
same wire.
VoIP
Voice over Internet Protocol
The use of the Internet Protocol (IP) for transmitting voice
communications. VoIP delivers digitised audio in packet form and can be
used for transmitting over intranets, extranets, and the Internet.
2
1
General Information
Witness Systems provides a business-driven multimedia recording, performance analysis and
e-learning management solution that enables companies to:




2
Enhance the quality of their customer interactions
Achieve greater customer retention and loyalty
Develop staff
Generate additional revenue
Main Contact Information
Main Location
Additional Location
Additional Location
Address
Corporate Headquarters
300 Colonial Center
Parkway
Roswell, GA 30076
USA
European Headquarters
Kings Court, Kingston
Road
Leatherhead
Surrey, KT22 7SL
Asia Pacific Headquarters
Room 1809-1818, 18F
Shui On Centre
6-8 Harbour Road
Wan Chai
Hong Kong
Main
Telephone
Main Fax
General E-mail
1-800-494-8637
+44 (0) 1372 869 000
+852 3198 6000
7707541888
support@witness.com
+44 (0) 1372 869 000
+852 3198 6100
3
Witness Contacts
EMEA
Name
USA
Leatherhead
Asia Pacific:
ANZ (within
Australia)
(elsewhere)
RoAP (rest of
APAC)
Phone
1 800 494 8637
0800 496 1111
Email
support@witness.com
cic-emea@witness.com
support@witness.com
1 300 367 394
support@witness.com
61 2 9232 5833
+852 8103 0104
support@witness.com
support@witness.com
3
4
Witness Support Contact Information
Team
Phone
Main Location (USA)
1 800 494 8637
Europe – UK
0800 093 1835 (out of
hours – for customers
with out of hours
contracts)
0800 496 1111 (7:00 –
19:00)
Fax
E-mail
Hours
770 754 1888
support@witness.com
Phone support is 24
hours a day from
3:00pm Sunday – 8:00
pm Friday (excluding
holidays); Off hours
(emergency based
support) is 8:00 pm
Friday to 3:00
Sunday.
01372 869 005
cic-emea@witness.com
Asia Pacific
1 300 367 394 (Aust)
61 2 9232 5833 (ANZ)
+852 8103 0104 (rest
of Asia Pacific)
During business hours
Out of hours pager
service available
N/A
N/A
N/A
The IPTelephony category requires 24x7 support contact. List all applicable after hours
numbers.
5
Witness Support Detail
5.1 General Information
Witness Systems will respond to customer telephone requests for assistance in the order
in which calls are received. For Standard Support telephone requests, Witness Systems’
intent is to answer its customer phone inquiries directly. If all Witness Systems CSR are
working with other customers at the time of a request, the customer calling for assistance
can expect to receive a return call from a Witness Systems CSR within 2 hours of the
original call to the company. A call directed to the Hotline number outside standard
support hours for critical issues will receive immediate attention and a return phone call
within 30 minutes.
In addition, Witness Systems also offer on-line support where customers can log in using
unique log in identification details to report and monitor progress of issue resolution.
4
5.2
Severity Description and Response Times
Priority/Severity Description
Current WITNESS® performance prohibits the
Critical
High
Medium
Low
5.3
Response Time
Immediate assignment of service
customer from conducting a critical business
resources to research and correct the
function, thereby creating an “emergency” situation problem as soon as possible.
With progress updates every 4 hours
Current WITNESS® performance has significant 5 business days.
impact on a business function, but the customer's With progress updates every day.
business is still functioning.
Current WITNESS® performance has a moderate Workaround in 15 business days and /
impact on a business function, but the customer’s or Next Service Pack
business is still functioning.
With progress update every 5 business
days.
Current WITNESS® performance has a minor
Incorporated into next release activities.
impact on the customer’s business operations.
With What’s New in release
documentation.
Support Level Definitions
Level
Bronze
Silver
Gold
Platinum
Description
09:00 – 17:00 Monday to Friday inclusive but excluding all Bank and
Public Holidays. Response time = 4 hours; On Site Support = 8 hours.
09:00 - 17:00 Monday to Friday inclusive but excluding all Bank and Public
Holidays. Response time = 2 hours; On Site Support = 4 hours.
07:00 - 21:00 Monday to Friday inclusive but excluding all Bank and Public
Holidays. Response time = 2 hours; On Site Support = 4 hours.
00:00 – 24:00 Monday to Sunday inclusive of all Bank and Public Holidays
Betw 07:00 –21:00 5 days Response time = 2 hours; On Site Support = 4
hours.
Betw 21:00 – 07:00 and 24 hr weekends Response time = 2 hours; On Site
Support = 6 hours
5
5.4
Call Flow Description
Initial Response
Queue/Pool Web
Once 'New Incident' is
accessed, the incident must
be worked. It cannot be
placed back into the 'New
Incident' queue.
Customer Communication
required to remove from
Initial Response/Pool Web
Inbound incident logged by call
handler/Tier 1/Initial Response and
placed in 'Initial Response and
Resolution Pool'
Initial Response
and Resolution
(Tier 1)
Ready for Closure
Closed
Field Engineering
Queue
Adtech Queue
Technical Analysis
(Tier 2)
Technical
Analysis
Incidents routed
by Manager to
Individuals
Technical Analysis Team
Engineering
Support
(Tier 3)
6
6
Witness Support Escalation Procedures
Elapsed Time
1-Hour
4-Hour
24-Hours
48-Hours
72-Hours
96-Hours
Priority 1
Priority 2
Urgent
High
CIC Manager CIC Manager
CIC Director/ES
CIC Manager
Manager
CIC Director/ES
SVP Global services
Manager
SVP Global
SVP
services
SVP Global
SVP
services
SVP
SVP
Priority 3
Medium
CIC Manager
CIC Manager
Priority 4
Low
CIC Manager
CIC Manager
CIC Director/ES
Manager
CIC Director/ES
Manager
CIC Director/ES
Manager
CIC Manager
CIC Director/ES
Manager
CIC Director/ES
Manager
CIC Director/ES
SVP Global services
Manager
For USA
Escalation Contacts
Title
Name Phone Number
Customer Interaction Centre
N/A 1 800 494 8637
Customer Interaction Centre Manager N/A 0770 7548644
Director Customer Interaction Centre N/A 678 254 5363
+44 (0) 1372 869 401
SVP Global Services
N/A - or +44 (0) 7767 791 599
For EMEA
Escalation Contacts
Title
Name Phone Number
+44 (0) 1372 869518
Customer Interaction Centre Manager N/A - or +44 (0) 7879 607 123
Engineering Support Manger
N/A +44 (0) 7767 791 581
Director EMEA CIC
N/A +44 (0) 7770 422 788
+44 (0) 1372 869 401
SVP Global Services
N/A - or +44 (0) 7767 791 599
+44 (0) 1372 869 445
SVP EMEA
N/A - or +44 (0) 7968 486 882
7
For Asia Pacific
Escalation Contacts
Title
Name
Phone Number
ANZ ProfSManager Terry Halvey +61 2 8907 0336
RoAP ProfS Manager Risk Israel
+852 3198 6226
ProfS Director
Ian Millward +61 2 8907 0330
SVP
Nigel Hewett +852 3198 6000
President
Nick Discombe +44 1372 869 468
7
Witness Product Information
8
Witness Product Overview
The Witness Systems Contact Store IP architecture provides 3 levels of products to suit
the different needs of Cisco customers to record and live monitor their IP telephony
conversations. These products are differentiated by functionality and the number of
simultaneous recordings they can handle. They are aimed at meeting different business
requirements of different user/customer groups. The following table lists the products and
related customer groups.
PRODUCT
eQuality ContactStore
Exec
eQuality ContactStore
IP Micro
eQuality ContactStore
IP
CUSTOMER
CEO’s, VPs – record an important call only.
As above, aimed at small call centre and bank level.
As above, with CTI and CRM integration with multi-site
support and application integration.
eQuality eWare and eQuality Unify are elements to the product suite that are embedded
in the product that manage the database and external CTI interfaces respectively. They
are not exposed to the customer unless a large enterprise solution is implemented.
9
Witness Product Features
See table below
10 Witness Product Performance (as tested)
The Witness Systems’ recording suite of software combines scalable platform
architecture to provide functionality to customers to suit their needs and budget. The aim
of the suite of software is to allow a customer an entry level solution that can be upgraded
without the need to replace any element of previously purchased software.
The following table shows the different performance capabilities of each level of the
suite.
8
Product
Capabilities
Contact Store Exec

Base System has server licence and bundled 2 channels of ondemand recording – 'Contact Store Exec' extension licenses.

The system has a maximum capacity of 30 Extensions.

Bundled is a single concurrent Eyretel Insight Viewer client license,
which cannot be expanded.

Multiple user accounts for the single license is possible.

No live monitoring is available; this requires an upgrade to Eyretel
Observer.

No full-time recording is available; this requires an upgrade to
Contact Store IP Micro
Contact Store IP Micro
Contact Store IP
Capabilities as Contact Store Exec, but also including:

Base System has server licence, database application and
bundled 1 Eyretel Insight Viewer client license.

Support for 1-30 channels of full time recording or Contact Store
Exec

Multiple Contact Store IP Micro recorders cannot not exist in the
same network, upgrade to Contact Store IP is needed

Single Eyretel Insight Viewer client license is expandable to 5
concurrent licenses

Any of the recorded channels can be live monitored with the
purchase of an Observer client license. The maximum live
monitoring of conversations with Observer is 5.
As Contact Store IP Micro, but also including:

Base System has server licence, database application and
bundled 5 concurrent Viewer client licenses.

Support for SQL Server 2000 Enterprise Database

Support for Multiple Recorders forming an enterprise Recording
network

Support for 1-60 channels of full time recording per server

Support for Replay Studio cal analysis solutions

Support for QualityCall Call Evaluation suite

Support for Performance Manager

Support for 3600 Busy Hour Call Attempts (BHCA)

Each recorded channel can be configured as a full-time recording
or Contact Store Exec channel

Any of the recorded channels can be live monitored with the
purchase of an Eyretel Observer client license. The maximum live
monitoring of conversations with Observer is 5.

Minimum machine specification is 2.4 Ghz pentium 4 processor, 1
Gb RAM; 250 Gb HD.

Consolidation time of 60 calls a minute.

Recording size of 8 Kb per second.
9
11 Witness Product Security Considerations
Witness Systems’ recorder system utilizes Windows 2000 Server and as such there is the
scope for an organization to undertake their ‘usual lockdown’ procedure for servers. The
scope and the level to which an organization secures their server(s) is varied and at the
discretion of that organization.
The main general area for securing the system lies in secure access to replay calls.
The users associated with particular extensions are set in the Administrator console and
the usernames themselves are derived from one of 2 scenarios: a) The Windows 2000
Local User accounts of the IP Recorder or b) The Domain/Active Directory users of the
Domain the system is hosted on.
Whether the system uses local users or domain users is normally decided by the size of
the system and organization. With 1 or 2 users needing to replay in an entire system it is
preferable to use standalone users to simplify the install, whereas more than 5 users
would mandate integration to existing domain users.
Achieving the different levels of security is purely an Internet Information Server setting.
12 Packaging, Bundling, Upgrading
The 3 levels of product are each licensed for specific capabilities and also maximum
number of concurrent recordings achievable.
Additionally the product is capable of being licensed on a timed-trial basis, i.e. 5-days,
where at the end of the trial the customer has the option of buying a full license key to
continue the use of the system in a non-destructive upgrade.
13 Overview of Integrated Solution (as tested)
At the highest level, the Witness Systems’ recording system consists of three conceptual
sub-systems:



The Capture subsystem, which captures information in real time.
The Storage subsystem, which compresses and retains these recordings for use later.
The Exploitation subsystem, which lets you exploit the recordings in your business.
These are shown in Figure 2-1 along with the major data flows through the system. In
many cases, these three components are built into a single PC often just referred to as a
‘recorder’ but in larger systems, the three components may be split across your entire
enterprise.
10
CAPTURE
EXPLOIT
STORE
Figure 2-1 Witness Systems Generic Recording System
The most effective way to ensure normal operation of each of these subcomponents is to
test the ‘exploit’ section, as for this to work, the other two must by definition be working.
To do this, simply record a call and attempt to replay that call and check the audio
quality.
Contact Store IP performs replay via a web-browser based application (Viewer) that can
be used from Internet Explorer V5.5 (Media Player 7.x).
14 Network Designs/Topologies (as tested)
14.1 Sample network designs/topologies
If the Cisco CallManager topology and recording requirements fall within any of the
standard scenarios described in Section14.2.1, configuration is straightforward and the
options are detailed here.
If the required system does not match one of these simple configurations, you will need
to review the information described below to ensure that an appropriate topology and
system sizing is determined. This is generally isolated to larger telephony sites where the
permutations and combinations are too varied for the scope of this document.
Contact Store IP is licensed according to the maximum number of concurrent extension
recordings that the recorder will support so, in all cases, you must determine how many
concurrent extension recordings are required.
14.1.1 Simple Topologies
The following commonly met scenarios are easily met by the configurations and
topologies described in the following paragraphs:
11
14.1.2 Single Site, Single Gateway and E1/T1 Trunk
Figure 1-1 Single Site, Single Gateway and E1/T1 Trunk
The system matches this category if:

Only calls originating or terminating in the PSTN are recorded.

There is a single Gateway with a single T1 or E1 trunk over which all such calls are routed.

The CallManager is connected to the same Catalyst switch as the Gateway.

IP phones may be connected to this or other switches via routers.


The Contact Store IP recorder is connected to a port on the same Catalyst switch as the
Gateway and CallManager.
All desks to be recorded use IP phones as opposed to SoftPhones.
If the system meets these criteria:

Configure the port to which the recorder is connected to SPAN both the CallManager’s port
and the Gateway’s port.

Select a PC of specification at least equal to the ‘Minimum recommended’ entry in Witness
Systems’ IP Pre-requisites document.
14.1.3 Single Site, Dual E1/T1 Trunks
Figure 1-2 Single Site, Dual E1/T1 Trunks
12
If the required system meets all the criteria of paragraph 14.1.2, except that there are 2 E1
or T1 trunks connecting the Gateway to the PSTN, adopt the same configuration as above
but ensure the PC specification is at least equal to the ‘Preferred’ entry in IP Prerequisites document.
14.1.4 Hub and Satellite Sites
Figure 1-3 Hub and Satellite Sites
The system matches this category if:

Only calls originating or terminating in the PSTN are recorded.

A single Router links the Satellite site to the central hub site.

IP phones are to be recorded at one or more satellite sites.

The Contact Store IP recorder is connected to a port on the same Catalyst switch as the
Router to the Satellite site.

All desks to be recorded use IP phones as opposed to SoftPhones.

The link to the remote site is 8Mbps or less.

No more than 60 concurrent calls will be carried over the link between central and satellite
site.
If your system meets these criteria:



Configure the port to which the recorder is connected, to SPAN the port through which all
traffic to the satellite site travels.
Select a PC of specification at least equal to the Preferred entry in the Witness Systems IP
pre-requisites document.
Size the disk storage accordingly.
13
Where there are multiple small satellite sites, a single recorder may be set to SPAN
multiple sites provided the total bandwidth does not exceed that stated above.
In sizing this topology you need to determine the maximum concurrent number of
external calls that can be placed from the remote site. This is often determined by the
bandwidth of the link to that site and is typically very much lower than the total number
of IP phones at that site unless the site is being operated as a Call Centre, in which case
up to 100% load must be catered for.
If a single router is used both to route traffic to the remote site and as a PSTN gateway,
then calls made from that site that go via the gateway will not be visible to the central
switch. In such cases, you must reconfigure your network to ensure that all calls are
transmitted via the port being SPANned.
14.1.5 More Complex Scenarios
If the topology and requirements do not meet one of the more common scenarios
described, you will need to determine the system sizing. You should determine the
information detailed in the paragraph below:
Recording
First determine whether you need to record:

Internal calls – within the CallManager system.

External PSTN calls – between one party on the CallManager system and a distant party on
the far side of a gateway.

External IP call – between one party on the CallManager system and a distant party on the
far side of a firewall and ISP up-link.
For each of the above call types determine which transmission types (G.711 or G.729A)
are to be used and in what proportion. Moving over to G.729A entirely will require much
less processor power on the recorder.
You should also determine how many concurrent calls of each of the above types will be
occurring and, of these, how many must be recorded.
Call Type
%
G.711
%
G.729A
Max concurrent
Calls
Max concurrent
Recordings
Internal
External
PSTN
External IP
14
15 Witness Product Requirements
To install eQuality Contact Store IP you will need the following:













A working Cisco CallManager system (Version 3.3) with at least one pair of IP
phones.
A non-switching Ethernet hub or a switched hub or router that supports Switch Port
Analyzer (SPAN).
Ethernet drop lead for the computer that is to be used as a recorder.
A high speed computer with Windows 2000 Server installed.
A single processor is required, due to limitations of Windows 2000, whereby the
packet driver cannot run with more than one processor.
Windows 2000 Service Pack 3.
Java Development Kit (version 1.2) installed on the pc where you are installing
eQuality ContactStore IP.
If eQuality Viewer is to be installed, you will need to install Microsoft Message
Queue and IIS.
A PCI Ethernet Network Interface Card (NIC).
Windows 2000 drivers for the Ethernet Card if not pre-installed.
SQL Server 2000 Installation kit.
eQuality Contact Store Installation CD.
Witness Systems licence code for the required channel count.
At least 6 gigabyte (Gbyte) free disk space for installation of the Calls folder.
You must be logged in as a local system administrator on the computer on which you are
installing this software.
15
LAN configuration
For VoIP recording to work, it is essential that the recorder can intercept the call control
packets passing between the IP phones and the call managers, and the voice streams
passing directly between the two parties engaged in each call.
For a Cisco implementation, you are recommended to use the SPAN algorithm to set up a
port on the Ethernet switch to receive copies of all the packets needed. Refer to the
System Administration Guide, for guidance on topologies.
Refer to the Ethernet switch documentation to set this up.
Platform Installation
To prepare the computer ready for the installation of the Witness Systems components,
the following steps are required:




Disable all power-saving features.
These could interfere with the recorder’s ability to start recording at any time.
Install correct Operating System
Windows 2000 Server must be installed and Service Pack 3 must be applied.
Windows 2000 upgrades are available as an option on the installation CD.
Install and set up Internet Information Server (IIS)
Ensure that IIS has been installed as part of your Windows 2000 Server installation
and that all necessary IP configuration has been undertaken to allow the PC to act as
a web-server.
Install Message Queuing Service
These services can be installed by the Message Queuing Service Windows
Component
Install Ethernet Card and Driver
Install the Ethernet card, checking for resource conflicts. Prove connectivity over TCP/IP
to the outside world. Ensure the PC has a valid IP address (can be statically assigned or
via DHCP).
16 Witness Product Installation Procedures (as tested)
Please refer to eQuality ContactStore IP Installation Guide document
17 Witness Product Isolation Procedures
18 General Witness Product Considerations
18.1 Limitations of this release
Please read the following limitations before submitting problem reports or enhancement
requests.

Confirm that the PC you are using is capable of sustaining recording under peak load
conditions. Guidelines are given earlier but PCs can vary considerably in
performance and Witness Systems recommend that you confirm loading figures on
your system.
16








Liability of Witness Systems for failure to record any calls is limited under the terms
of supply.
A limited range of PC configurations have been tested to date. Configurations
recommended at this time will include significant contingency - which may prove
over cautious in the medium-term.
A limited range of compression formats are supported.
This list will grow and if an as yet unsupported format is to be received or is required
as the long term storage format, please contact Witness Systems for the latest
position.
Configurations above 90 concurrent recordings, that is, requiring multiple recorders
or separate database/disk servers must be designed and verified by Witness Systems.
It should be noted that only calls between extensions marked for recording will be
recorded. Therefore, if a call is transferred from a recording extension to one which is
not recorded, recording will cease.
For example, when A calls B, who forwards to C, if B only is marked for recording,
then the forwarded call (from A to C) is not recorded.
If Executive Record is enabled, even though there is no active call, the phone
displays the message:
Call is being recorded
You can clear this message by selecting the Exit button.
If you have the options Record all calls and Executive Record enabled, please note
that Record all calls will override Executive Record.
For example, a call between a marked extension and an executive extension will be
recorded.
If Executive Record is enabled and a call is being recorded, if another incoming call
arrives, causing an interrupt, then the recording is stopped and Executive Record has
to be manually enabled again when the original call is retrieved.
18.2 If you alter your server
If you make any changes to the setup of your server you should take a ghost image of it
to allow it to be reconstructed quickly in the event of major hardware failure.
18.3 If you alter your network
Contact Store IP can only record calls whose voice packets are presented to the port(s)
connecting it to the network.
The recorder also has a finite capacity, both for filtering required calls out of the
incoming packet stream and for the number of calls that it can record concurrently.
Any changes you make to your network topology can potentially impact the recorder’s
performance and you should follow these guidelines whenever you change your network.
Changes to IP addresses
If you change the IP addresses of any components of the system, you should perform a
full commissioning test to confirm correct recording of all required calls.
17
Changes to network topology
If you change the topology of your network you should check all SPAN port settings and
then perform a full commissioning test to confirm correct recording of all required calls.
18.4 If you alter your IP telephony system
Adding new telephones
If you add additional telephones you should check SPAN port settings and the
appropriateness of existing tap points, adding additional taps as required.
You should then confirm that calls to and from these new phones are recorded correctly.
Where fall-back IP-PBXs or clusters are used, you should force the new phones to use
each server in turn and confirm that calls are still recorded. It is possible, for example, to
have a backup server that is not correctly configured in the required VLAN. Such errors
can go unnoticed for long periods until the time when the main server fails and operation
switch to the standby server.
You should also confirm that the total loading on the server is still within its capability.
Adding additional gateway capacity
Follow the same processes as for additional telephones:




Check SPAN settings
Confirm recording of calls via these ports
Confirm recording under failover scenarios
Confirm load is acceptable
In this case, traffic tends to increase gradually after additional capacity is installed, hence
you should pay special attention to loading over the following few weeks until it
stabilises.
Configuring your hard disk
Your hard disk must be configured to have a minimum space of 30,000. You use the
System Administration tool to configure this.
PROCEDURE
1 To do this, from the System Administration tool, expand the tree, locate and click on the
ContactStore Storage Processing node, then in the right pane, check the setting against the
Minimum space to maintain entry.
2
Ensure the value is set to 30,000.

There is no deletion setting. When your disk becomes full, it will automatically delete.
18
19 Troubleshooting
19.1 Debugging and Diagnostics
Should you experience problems with your Contact Store IP recorder, there is a wide
range of diagnostic tools available to help you determine the cause of and ultimately
resolve the problem. In summary you have at your disposal:

Several components of the Windows 2000 system management tool set are of use. Most can
be accessed via the Microsoft Management Console (MMC) snap-in provided. These
include:









Windows Task Manager. To see if the appropriate tasks are running.
(The following tools have been included in the default console settings of the
Administrator tool for your convenience).
Windows Service Manager. To allow you to stop and start services manually.
Windows Performance Monitor. To observe a wide range of performance and
diagnostic counters – this can directly accessed from within the Administrator
console under the system monitor branch.
Windows Event Log Viewer. To observe any service start/stop/error messages
generated by the Contact Store services.
In addition, each Contact Store service outputs more detailed diagnostic information into log
files accessible via the Administrator console.
You may also choose to run each service from a command-line prompt, giving more
immediate, real-time visibility of any major events occurring.
Witness Systems’ CTI Studio can be used to observe and modify the behaviour of the Unify
service as it communicates with the other components in the system.
You can also use all of the SQL Server administration tools to examine the contents of the
database as recordings progress.
19
19.2 Error Messages
The following are a list of known error messages and issues.
eQuality Contact Store service(s) fail
Check the Application Event log, see example below, for error messages using the
Windows Event Viewer or the MMC snap-in provided.
Shut the service down and run as console application with the –c command-line option.
Also check for the log files Capture.log, Compressor.log, Consolidator.log and
DiskManager.log in <installation dir>\log files (The default directory is C:\Program
Files\Eyretel\Contact Store IP\Log files).
If you run one of the executables from the command line, the log file will be located in
the path from which you ran the executable.
If the Contact Store services will not start, check the PC’s node name. Problems have
been observed when the computer's nodename (as recognised by SQL server for
example) differs from that shown in the properties of the LAN connection and in the
Network Identification tab of My Computer properties.
Solution:
Only solution found to date has been to reset the Network node name to match the
underlying name.
Calls observed but no audio
Solution:
Check the network topology carefully. If using a dumb, non-switching repeater type hub,
beware of mixing 10 and 100Mbps ports as this tends to stop RTP packets from being
received reliably.
If in doubt, plug the recorder PC into the back of an IP phone from which calls are being
made. If the recorder is functioning correctly, it will record calls to/from that phone.
Service functions but no performance counters visible
Solution:
Confirm that the service is running under the user account EYR_Contact7k. This account
should have administrator privileges.
20
You may also find that the SNMP service was running when the installation program
tried to install the performance monitor variables. The current installation kit tests for and
warns if this service is running but as it relies on the user to stop the service, it may not
succeed in installing the performance monitor variables. The only option here is to
remove eQuality ContactStore IP and reinstall with SNMP service stopped.
The performance counter can also equal zero if you are viewing them through terminal
services.
Configuration under Tools/Options fails
When setting the eWare details a dialog box stating ‘Error in setting the Eware2
Directory’ is displayed.
Solution:
PROCEDURE
1
Double check the password entered for User EWCODE in SQL Server.
2
Create the directory C:\Program Files\Eyretel\Eware2\Log Files and try again.
3
Examine the log file produced in this directory.
Cannot search for calls
Solution:
Confirm that the databases ‘EwareCalls’ and ’EwareConfig’ are present within the SQL
Server.
If call details are visible within the SQL server’s EwareCalls Database and call files are
appearing in the correct call path, but cannot replay calls successfully in Vision, try the
following:
Search Manager setup
Confirm the registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\EYRETEL\Search Manager\Config\Path
is set to the node name of the PC running the SQL server as the example shown below.
Do NOT include ‘\\’ at the start.
Ignore other settings shown above.
21
Missed call, audio out of sync, missing call party identification information
Cause:
These types of issues can be caused by the use of anti-virus software. Use of any antivirus software will cause server performance issues, in particular packet loss.
Solution:
We recommend if you need to run anti-virus software, that you use Norton AntiVirus
version 7.5 or greater and make the following changes:




Run with File System Realtime Protection scanning disabled.
Schedule scanning to occur outside of recording hours and avoid full system scans.
Exclude .DAT, .TMP, .XML and .WAV files from scans.
Exclude the following paths
 C:\Calls
 C:\Program Files\Witness Systems\ContactStore\PersistQ
 C:\Program Files\Witness
Systems\ContactStore\ArchiverTemp
 C:\Program Files\Witness Systems\Unify\PersistQ

These 4 paths assume the default installation path, if you set custom paths during the
install, you will need to exclude these.
19.3 Tools / Aids
Performance logs and alerts
This branch gives you access to the standard Windows Performance logs and events
utility. You can use these branches to set up regular monitoring and logging and to alert
you to system events and exceptions such as lack of disk space, overload etc.
Counter logs
You can use this branch to log periodic samples of performance monitor variables.
Trace logs
You can use this branch to log system events such as process creations and deletions.
Alerts
Using the Administrator’s System Monitor node, you can observe, via Microsoft’s
Performance Monitor, the behaviour of the Contact Store IP system. A default view is
provided which shows all the variables that are available. To alert you to incorrect
operation or system problems, however, the system uses a range of system alerts set
through this branch of the Administrator.
A number of triggers are also defined which indicate abnormal performance. You may
wish to customise these to suit your needs and/or alter the actions that will be taken
22
should the trigger fire. This can include generating SNMP traps, pager messages, emails
etc. Refer to the help function on Microsoft Management Console for instructions on how
to achieve these. Most properties can be set by selecting one of the alerts in the righthand pane and right-clicking it. Select Properties to adjust the threshold and/or actions to
be taken when the alert is triggered.
For example, you can select to have alerts pushed to a different PC from the server if you
wish to track problems from your desktop PC.
ALARM MESSAGE
PROBLEM
ACTION
Failed to Compress
Compressor unable to
compress audio files
Check .wav files are being produced,
disk is not full.
Failed to Consolidate
Consolidator unable to
upload call details into
database.
Check .xml files are being produced;
check database is running and
accessible
Disk Space Low
Disk Manager cannot
free disk space
Check for other applications using
excessive disk space.
Failed to Delete
Disk Manager unable to
delete files
Reboot to clear any erroneous file locks.
Check for read-only files in recording
path.
Excessive Memory Buffer
Usage
Overload causing
memory buffer to
backlog.
Increase CPU power or memory or
decrease traffic loading.
Losing Audio Packets
More than a few packets
per second being lost
Check network loss rates, check for
overload.
Excessive Packet Rate
Too many packets per
second being received
Reduce packet load through better
SPAN settings.
Concurrent Usage Nearing
Limit
Within 10% of allowed
concurrent loading.
Increase server capacity to full Contact
Store IP.
Archive Media Full
DVD-RAM disk is full
Change disk
Archive Drive Empty
No disk in DVD-RAM
drive
Insert and format clean disk in DVDRAM drive.
23
Services
This branch gives you access to the standard Windows Service control utility. You can
use this to start, stop and observe the status of the various services running on the
computer.
Of particular interest are the following services:






Contact Store IPCapture
Contact Store Compressor
Contact Store Consolidator
Contact Store Archiver
Contact Store Disk Manager
Unify
All of these should be running permanently.
On installation, all of these services are set to restart automatically on failure. You may
wish to have the system alert you should these services fail. Select the service required,
right-click and select Properties on the shortcut menu to change these recovery options.
19.4 List of Commands
Instead of running MediaStore.exe as a service, it can be run from the command line.
This is particularly helpful when first configuring the system as it gives immediate
feedback of calls being recorded in the command window,
When run from the command line, IPCapture.exe supports the following options:
-c run as console application rather than a service
-i install as a service (will fail if already installed).
-r de-install as a service
-f accept input from a test file rather than the network
When run with the -c option, each service will display text indicating when the service is
fully initialised and ready. Each of the services will then display the following service
specific information when operating correctly:
ContactStore
Service
Shows…
When…
Consolidator
Processing
<filename>
Unify instructs it to upload call
details to the SQL Server
Consolidator
DONE
Database record inserted
successfully
Compressor
Processing
<filename>
Unify instructs it to compress
.WAV file specified
Compressor
DONE
.WAV file compressed and
original deleted
24
ContactStore
Service
Shows…
When…
Disk Manager
Entering
X:\yyy\zzz\aa\bb
Files in the specified directory
are next candidates for deletion
20 Redundancy and Recovery
20.1 Redundancy
Utilizing standard Hardware platforms the resilience of the server itself can be dictated
by the customer and any Cost/Benefit analysis they derive. Witness Systems strongly
recommends that a minimum of RAID1 configured mirrored hard disks be deployed.
However for further redundancy it is possible to implement a distributed architecture to
the system. Following is a summary of some of the elements that can be added for
redundancy:
1. Multiple storage locations – replicate the audio and XML file created for each call to a
secondary device – i.e. Off site, SAN etc.
2. Utilize a centralized SQL Server Database.
3. Replicate & confirm backup of the Call details in the SQL 2000 Database – remember
that the underlying data for each call is still stored as an XML file and hence can be
recovered by Witness Systems’ Professional Service in the event of a total failure.
4. Connect to multiple switches each with the same SPAN algorithm allowing for
redundant feeds from the switches (note this would require the use of min. Adaptec
DualPort Ethernet card).
20.2 Recovery
Witness Systems’ IP Recording suite of products is totally passive to the AVVID
environment. Hence a failure of the system and subsequent automatic or manual recovery
has NO impact on the AVVID system.
20.3 Dual Spanning
To achieve maximum resilience and redundancy in the system, you can use double
spanning.
21 Interoperability Test Results
22 Cisco Product Information (as tested)
Cisco CallManager 3.3.
Cisco 7960 phones.
25
23 Cisco Product Interfaces (as tested)
Witness Systems’ Contact Store IP does not interface directly with to the Cisco Product
Interfaces. It passively decodes the SCCP messages.
24 Cisco Product Requirements (as tested)
The minimum requirements for the system are those needed for a basic Cisco
CallManager demo system, providing basic internal call functionality:


a working Cisco CallManager system V2.4 or higher
at least one pair of IP phones, 3 phones provide the ability to transfer calls etc.
 7960, 7940 & 7910 phones are supported
 Currently Cisco SoftPhone 1.1 is NOT supported. Although Softphone 1.2 is
supported.

a non-switching Ethernet hub or a switched hub that supports the “RMON SPAN”
algorithm.
 Optionally the Visual Switch Manager web administration tool for configuring
Cisco switches

2 Ethernet drop leads for the PC to be used as a recorder – one for SPAN connection of the
Voice VLAN and one for Data LAN connection to service Client replay requests.
25
CallManager Device Table
NO CallManager Devices need to be configured.
26
APPENDIX A
Patch
Number
Files Changed
Filename
CSIP0009
This is a rollup patch for
IP please read attached
readme.
See readme.txt
CVWR0016
EyrInfQueuedCache.dll
7.1.0.134
CVWR0017
CSIP0012
player.htc
actionflow.htc
replaymanager.htc
itemsetview.htc
Description
Version
The top (or near
top) line of the
player.htc,
actionflow.htc,
replaymanager.htc,
and
itemsetview.htc
files will read as
below:
<!-- Patch
CVWR0017 -->
IPCapture.exe
3.2.7.151
RTPHandler.dll
Replay can be very slow when a UNC
calls path is used instead of accessing
the Calls share/folder directly for an IP
capture platform.
This patch does not affect HTTP
retrieval. If HTTP interfaces are used
they will not be improved
If a user attempts to replay multiple
calls quickly and repeatedly the Contact
Viewer
browser application can get into a state
in which it does not replay subsequent
calls.
Sometimes it recovers but the user can
be in the situation where clicking a call
does
not result in a replay.
This patch improves the performance of
the system, and adds extra diagnostic
variables.
1.0.0.28
NGAComponent
7.1.0.3
Unify.spt
CSIP0014
Observer.exe
7.1.0.12
7.2.0.26
This patch fixes a number of issues in
observer:
- When an extension was monitored it
would be highlighted in green in the
calls view, but when that call was
terminated and a new call started on the
same extension the call would be
monitored but the call would not be
highlighted in green.
- The stop button would not work and
you would continue to hear audio even
after pressing stop!
- On a machine that has a full eventlog
Observer would crash with an error
message.
27