Avid iNEWS Command Best Practices
Failover and Redundancy
This document outlines the Command Server failover and Command hotswap failover functionality of
iNEWS Command. It includes the performance expectations, recommended setup, and failure/recovery
scenarios.
1 Performance Expectations
1.1 Failover
iNEWS Command 1.0.x
In the event of a network or power failure to the primary iNEWS Command Server, playout
functionality is expected to return within 30 seconds.
iNEWS Command 1.1.0 (Improved performance)
In the event of a network or power failure to the primary iNEWS Command Server, playout
functionality is expected to return within 15 seconds.
iNEWS Command 1.1.1 (Improved performance)
In the event of a network or power failure to the primary iNEWS Command Server, playout
functionality expected to return within 10 seconds.
1.2 Hotswap
iNEWS Command 1.0.x and Command 1.1.x
In the event of a network or power failure to an Airspeed device, playout functionality is
expected to return within 10 seconds.
2 Recommended Setup for Command Server Failover
2.1 Configuration
It is recommended to have all primary Command services on one designated primary server
with the backup services running on others. Command determines primary and backup roles
based on the length of time the service has been running. The server that has been running
longer is the primary. Therefore, in order to get all Primary services on one server, it is
necessary to start the primary server first and when all services have started, start the
backup servers. This is the only way to ensure all services on the Primary server take on the
Primary role. This allows for more consistent behavior during times when redundancy is
needed and helps simplify any necessary troubleshooting efforts.
All Command Servers should be assigned static IP addresses.
Windows Media Sense should be disabled. Microsoft QArticle 239924.
If the Command server has dual network cards, the unused connection must be disabled.
2.2 Redundant Services
It is best to dedicate a machine to be a Command server. We do not recommend running the
main Command services on a workstation, Deko, or Thunder.
The following are redundant Command services:
o AirSpeed device service
o Playlist Controller service
o Playlist Manager service
o Playlist Store service
Rev. B
Copyright 2008 Avid Technology, Inc.
o
o
Inventory Manager service
Avid MOS Communication Service: This service is server specific, meaning that services
on different computers act independently. In a failover scenario, the Command playlist
will become disconnected, so users can continue playout functions. To get an updated
playlist, the user must perform a MOS redirection in ENPS to the Command Server
running the now primary Avid MOS Communication service and then continue as before.
2.3 Non-redundant services
There are some services that are not redundant:
o Lookup Service – This service acts independently. If a second one is running, it is doing
the same work as the first Lookup service – they basically work as 2 primary services. In
a failover scenario, the second Lookup service is already working, and services would
continue seamlessly.
o iNEWS Communication Service – This service is server specific, meaning that services
on different machines act independently. In a failover scenario, the Command playlist will
become disconnected, users can continue playout functions. To get an updated playlist,
the user can reload the iNEWS rundown to the other Command server running the Avid
iNEWS Communication ‘;service and continue.
o Deko & Thunder Device Services – These services need to run on the Deko and Thunder
themselves, and you cannot install a second service on a machine.
o Service Starter Service, Machine Monitor Service, Time Synchronization – These
services are machine specific – redundancy does not apply.
2.4 Primary and Backup Roles for Command Services
Command Services can have two roles, primary and backup. There can be only one primary
service, but there can be several backup services.
When a primary service goes down, Command conducts an election, and changes the
backup service to primary. The new primary service will remain primary until it is stopped or
restarted. When the old primary service comes back up, it becomes the backup service. If the
Command Server running the primary services were to incur a failure or network disconnect,
all of the backup services running on the redundant Command Server would become
primary.
3 Failure/Recovery Scenarios
This section provides various failover scenarios, the symptoms seen at a Command
Workstation that would indicate such problems, and the recommended procedures for
recovery.
Failover Scenario
Primary device channel
failure.
Mirrored device channel
failure
Rev. B
Symptoms seen in
Workstation UI
Status bar indicator for that
device channel will show
disconnected and the
playability icon will change to
.
Status bar indicator for that
device channel will show
disconnected and the
playability icon will change to
.
Copyright 2008 Avid Technology, Inc.
Recommended Recovery
Procedure
The person running the switcher will
need to take the mirrored channel to
air.
No recovery needed
Failover Scenario
Symptoms seen in
Workstation UI
The channel status shows this
icon, , when the channel
fails, and when Hotswap takes
over, an Alert appears and the
channel status changes to this
If the Hotswap Channel is
needed due to device
channel failure
Power Outage or computer
restart on the server with the
Command services running in
a primary role.
, to indicate that it has
icon,
been replaced by the Hotswap
channel.
The Count Up and Count Down
clocks will stop immediately.
There will be multiple alert
messages sent to the
workstation saying that it can’t
find the Playlist Store and other
services. Note that the
currently playing video will
continue to play, but there will
be no transport control.
Power Outage or computer
restart on the Command
server running the backup
services.
iNEWS loses its connection to
the Avid iNEWS
Communication service
where iNEWS monitor is
currently loaded.
Nothing seen
Stop individual primary
Command service (one of
Playlist Controller, Manager,
or a Device service).
The user may see an alert in
the Alerts panel and there may
be a short playout interruption
while the backup service takes
over the primary role.
Rev. B
The iNEWS client will show
monitor in an ERROR state.
Command will show the playlist
as disconnected (red bar
across top of playlist view).
Any updates to the iNEWS
rundown while in this state will
NOT be reflected in Command.
Copyright 2008 Avid Technology, Inc.
Recommended Recovery
Procedure
The person running the switcher will
need to take the hotswap channel to
air.
Within 10 seconds, you will receive
an alert message indicating that the
Playlist Controller is available, with
the details stating that items can be
played again. If a newsroom playlist
was being controlled at the time of
the failure, the disconnected rundown
can be played before the rundown is
reloaded to the redundant iNEWS
Command server (now running
primary services).
No recovery needed.
iNEWS monitor will need to be
reloaded to the Command server
running the secondary Avid iNEWS
Communication service. After it’s
been reloaded to Command, unlock
the channel groups in the
disconnected playlist (when
convenient) and lock them for the
reloaded playlist and continue playing
to air.
Wait until the workstation returns to a
normal state and then continue.
Failover Scenario
Loss of network connectivity
on the server with the
Command services running in
a primary role.
Symptoms seen in
Workstation UI
The Count Up and Count Down
clocks will stop immediately.
There will be multiple alert
messages sent to the
workstation saying that it can’t
find the Playlist Store and other
services. Note that the
currently playing video will
continue to play, but there will
be no transport control.
Important information
pertaining to network
restoration:
There may be a scenario where
another 10 second loss of
playout control may be
experienced when network
connectivity is restored.
Recommended Recovery
Procedure
The services running in a backup role
will take over as primary and restore
functionality within 10 seconds. If a
newsroom playlist was being
controlled at the time of the failure,
the playlist will need to be reloaded to
the redundant server (now running
primary services) from the newsroom
system.
Important information pertaining to
network restoration:
If possible, have someone restart that
server before network connectivity is
restored. If not, Command will go
through another election and the
Command workstation user will incur
another 10 second period where they
don’t have playout control when the
network is reconnected. Once the
show is over, the system
administrator may want to restart
services so the primary server is
once again the primary server.
Loss of network connectivity
on the server with the
Command services running in
a backup role.
No effect when network
connectivity is lost.
No effect when network connectivity
is lost. Let Command automatically
recover if the network connectivity is
restored.
ENPS loses connectivity with
the primary Avid MOS
Communication service
Command will show the playlist
as disconnected (red bar
across top of playlist view).
Any updates to the ENPS
running order while in this state
will NOT be reflected in
Command.
Nothing seen
In ENPS, perform MOS redirection to
the Command server running the
redundant Avid MOS Communication
service.
ENPS loses connectivity with
the Avid MOS
Communication service
running in the backup role.
No recovery needed.
4 Troubleshooting Scenarios
This section provides miscellaneous troubleshooting scenarios and the recommended
procedures for resolution.
Troubleshooting Scenario
Rev. B
Recommended Resolution Procedure
Copyright 2008 Avid Technology, Inc.
Troubleshooting Scenario
Recommended Resolution Procedure
Failed communication
between Interplay Framework
and Command services.
Ensure that the DNS entries are accurate and have reverse lookup
enabled. Also verify that your TCP/IP settings have the correct primary
and alternate DNS addresses.
Dual network card enabled.
Avoid using host names.
You must disable the unused connection.
Rev. B
Copyright 2008 Avid Technology, Inc.