Best Practices Guide
BEST PRACTICES FOR INTEGRATING
TELESTREAM VANTAGE WITH
EMC ISILON ONEFS
Abstract
This best practices guide contains details for integrating
Telestream Vantage workflow design and automation
software with the EMC Isilon OneFS 6.5 file system.
July 2013
Copyright © 2013 EMC Corporation. All Rights Reserved.
EMC believes the information in this publication is accurate
as of its publication date. The information is subject to change
without notice.
The information in this publication is provided “as is.” EMC
Corporation makes no representations or warranties of any kind
with respect to the information in this publication, and specifically
disclaims implied warranties of merchantability or fitness for a
particular purpose.
Use, copying, and distribution of any EMC software described
in this publication requires an applicable software license.
For the most up-to-date listing of EMC product names, see
EMC Corporation Trademarks on EMC.com.
EMC2, EMC, the EMC logo, Isilon, OneFS, InsightIQ, SmartCache,
and SmartConnect are registered trademarks or trademarks of
EMC Corporation in the United States and other countries.
All other trademarks used herein are the property of their
respective owners.
Part Number H10533.1
Best practices for integrating Telestream Vantage with EMC Isilon OneFS
2
Table of contents
Introduction ................................................................................................... 4
Typical Vantage domain configuration ........................................................... 4
Isilon cluster design ....................................................................................... 6
Telestream Vantage service configuration ..................................................... 7
Telestream Vantage management .................................................................. 8
About EMC Isilon .......................................................................................... 10
Best practices for integrating Telestream Vantage with EMC Isilon OneFS
3
Introduction
Telestream is the leading provider of enterprise-class video transcoding and
workflow software. The world’s leading media and entertainment companies trust
the combined Telestream and EMC® Isilon® solutions to automate the production
of content for broadcast, video on demand, IP television, cinema, and mobile devices.
The complementary Telestream Vantage® and EMC Isilon scalable architectures easily
adapt to constantly changing workflows. Servers are added to a multiserver Vantage
domain to meet the industry’s increased demand for transcoding, analysis, metadata
processing, and quality control services as needed. As the bandwidth and storage
requirements for Vantage workflows increase, EMC Isilon scales to meet these
demands. EMC Isilon customers add storage nodes to linearly increase bandwidth
and capacity, without introducing downtime and the risk of missing a critical content
delivery deadline.
This best practices guide details how EMC Isilon OneFS® 6.5 and Telestream Vantage
are combined to rapidly deploy new media workflows. EMC Isilon clustered storage
utilizes standard storage protocols like SMB, NFS, FTP, HDFS, HTTP, and Object APIs.
The added cost and infrastructure constraints of a Fibre Channel storage area network
are no longer an issue when scaling workflow automation. Telestream follows common
IT data center best-practices in their own documentation for deploying Vantage in a
NAS environment. Compared to other media and entertainment applications, there is
very little performance tuning or custom configuration required when integrating
Telestream Vantage with EMC Isilon OneFS.
Typical Vantage domain configuration
The Telestream Vantage application consists of a central Microsoft SQL Server
database and an array of servers running the distributed Vantage services. The
database tracks a queue of automated workflow actions that are allocated to the
array of servers running the Vantage services. Vantage workflow actions include
media content processing features, such as transcoding, ITU 1770 audio loudness
correction, quality control, metadata extraction, media format analysis, file delivery,
fingerprinting, and logo insertion.
Queued actions that become active are intelligently distributed to the best available
server in the array, based on a comparison of workflow status and server resource
availability. The Vantage array configuration is tuned to run enough concurrent jobs
on each server that the CPU utilization stays around 80 percent on every core of
every server running the Vantage services. If the Vantage servers have been
configured to run at greater than 80 percent CPU utilization, there is a risk of
temporary service failure and reduced workflow performance.
Transcoding is a CPU-intensive action. Typically, transcode workflows decode from
a mezzanine-quality media compression format, like MPEG 2, and encode to a highly
compressed format, such as H.264. The most common bottleneck in these workflows
is the speed at which a decoded file may be encoded to new media formats before
running out of CPU resources. As a result, it is common for a single 12-core Vantage
server running concurrent transcode jobs to generate less than 100 megabytes per
second of read access while decoding, and less than 10 megabytes per second write
access while simultaneously encoding to multiple output formats.
Best practices for integrating Telestream Vantage with EMC Isilon OneFS
4
While the moderate bandwidth of a typical Vantage server configuration is easily
handled by a non-blocking Gigabit Ethernet switch like the Cisco 3750, workflows are
like snowflakes—they are never identical. A workflow that decodes from an uncompressed
media format and encodes to another format with a low CPU cost may demand a multihomed configuration with a dedicated 10 gigabit private network when connecting to
network-attached storage. It is possible for a single 12-core Vantage server to
generate up to 400 megabytes per second of read traffic in a transcode workflow
when processing multiple high-definition media assets. Vantage is tuned to assign
different weights to different servers in its framework. Thus, a newer server with
faster CPUs is added to the array and the array is tuned to prefer the faster server
when allocating new CPU-intensive actions. In a typical installation of Telestream
Vantage with EMC Isilon clustered storage, workflow queue depth and wait time are
the metrics tracked to determine when additional compute or bandwidth resources
are required.
Figure 1. Typical Telestream Vantage server installation diagram
For more information on planning a multiserver Vantage installation, please see the
Telestream Vantage Installation Guide.
Best practices for integrating Telestream Vantage with EMC Isilon OneFS
5
Isilon cluster design
Since the Isilon network-attached storage platform is not hindered by a single “head”
for scaling network throughput, there is no need to size OneFS 6.5 performance for
a worst case scenario. OneFS 6.5 scales to meet changing bandwidth and capacity
requirements by simply adding nodes to the cluster.
When sizing an Isilon cluster for a new installation, Vantage server performance is
benchmarked using local storage in order to determine bandwidth requirements for
a single server in the array at 80 percent CPU utilization.
After determining the required workflow throughput requirements for a single server
in your Vantage installation, work with an EMC Isilon sales representative to determine
the server to storage node ratio required for your environment. The Telestream Vantage
workflow is well-suited for the EMC Isilon X200 series storage node. The X200 hardware
configuration may be tailored to meet your particular workflow requirements by specifying
drive capacity, adding 10 Gigabit Ethernet, and increasing RAM. The X200 may be
fitted with SSD drives to store some or all of the file system metadata, a feature that
is useful when working with digital intermediary and broadcast graphics workflows
with image sequences consisting of thousands of DPX or TARGA files. If you require
accelerated read performance for image sequence files, contact your EMC Isilon support
representative about the performance tuning settings for pre-fetching image
sequences into cache.
The most bandwidth-intensive actions in most Vantage workflows are simple FTP or
SMB2 file movements. For example, a transcode workflow that generates an output
format to be used on a broadcast video server can include an FTP task to move the
file from Isilon to the video server’s local storage.
Given the highly distributed nature of the Telestream Vantage server array, the
default OneFS file system configuration is optimal for most workflows. EMC Isilon
OneFS directories used by Telestream Vantage work well with “Concurrency” set as
the data access pattern, SmartCache-enabled. A new cluster expecting growth from
an active transcode workflow should use a data protection level of 2:1, meaning the
file system is protected against the failure of either an entire node or two drives from
any nodes in the cluster. These settings are the file system defaults for OneFS 6.5.
File system parameters such as parity, SSD metadata acceleration, and data access
patterns may be customized at the folder or file level, regardless of the default
file system settings, to meet different application performance or data protection
requirements. Please refer to the OneFS 6.5 User Guide for more information
on file system settings.
Since OneFS 6.5 is a single file system designed from the ground up to handle
massively concurrent file access, there is little need to move files in a transcoding
workflow utilizing the Telestream Vantage API for workflow orchestration. Large
Telestream Vantage workflow automation environments often utilize source media
managed by a central media asset management application. OneFS 6.5 is designed
to be the central repository for ingest, media asset management, editing, and archive
workflows. A proper media production workflow design utilizes a common share for
media assets that is available as a read-only volume to all users in the media production
workflow, and as a read/write volume to the group of automated services responsible
for media asset management, content storage management, and ingest. This design
Best practices for integrating Telestream Vantage with EMC Isilon OneFS
6
ensures that the managed assets are only deleted or modified through an auditable
media asset management action that can be traced back to the responsible service
and user requesting the service.
Without NTFS access control lists to enforce read-only access to a central repository
of managed assets, the workflow is at risk of accumulating “orphaned” media that is
not tracked by asset management. The workflow is also at risk of generating invalid
asset records in the media asset management database due to unmanaged file deletion
or modification. Properly planned NTFS access control list creation greatly reduces the
need for resource-intensive processes that constantly troll terabytes to petabytes of
storage to validate the integrity of your media asset management database against
physical storage.
In current media asset management workflows, the media files behind managed
assets are never modified. Managed media assets are either read by a nondestructive
process that saves changes to the media in externalized project metadata and “render”
files, as is the case with nonlinear editing workflows, or, as is the case with transcode
workflows, a source file is often read, decoded to RAM, and encoded back to a new
file in a different location than the source file. Transcode services write derivative files
to an EMC Isilon OneFS folder with NTFS access control lists permitting write, modify,
and delete access rights for both the Vantage services and any external services used
in the content delivery pipeline.
OneFS 6.5 intelligently moves files into a globally coherent cache built from the RAM
of every node in the cluster, eliminating the risk of disk contention when multiple clients
access the same file asset concurrently from multiple workflows. The capacity of this
cache is not bound to a single storage “head” as it is with other network-attached
storage solutions. On a five-node cluster in which each node contains 24 gigabytes of
cache, there is 120 gigabytes of RAM that a single file may be cached in for concurrent
read access. When the RAM in each node of the cluster is upgraded to 48 gigabytes,
the coherent cache of the cluster is upgraded to 240 gigabytes. The cluster can be
expanded without disrupting the Vantage workflow by adding five more nodes,
bringing the cluster coherent cache to 480 gigabytes, and so on.
Telestream Vantage is rarely the only service accessing storage in an automated
transcoding workflow, so you should allocate bandwidth for ancillary applications such
as content ingest and streaming playback. Consider the delivery method for the output
media when designing an EMC Isilon OneFS clustered storage environment. The EMC
Isilon OneFS operating system offers native support for the industry-standard Aspera
fasp™ high-performance file-transfer server software used by content distributors
around the globe.
Telestream Vantage service configuration
For best performance, install Telestream Vantage on the Microsoft Windows Server
2008 R2 operating system and use EMC Isilon OneFS v6.5.4 or greater. Windows
Server 2008, Vista, and Windows 7 will automatically negotiate an SMB2 connection
with EMC Isilon OneFS v6.5 and above. Thanks to improvements, such as support for
multiple SMB commands within a single packet and larger buffer sizes, SMB2 performance
is more than twice that of SMB1 over Gigabit Ethernet and three times greater over
higher-latency WAN connections.
Best practices for integrating Telestream Vantage with EMC Isilon OneFS
7
While Vantage works with SMB, FTP, and HTTP as file transfer protocols, any file
location that is to be used for a transcoding or analysis action must be configured
to use UNC paths to an SMB share in order to process media. SMB shares should be
entered in the Vantage Management Console or Vantage Workflow Designer in the
format \\server\share, where \\server is the fully qualified domain name of an
EMC Isilon SmartConnect™ zone using static IP allocation, and \share is an SMB share
on said EMC Isilon cluster. SMB shares mapped as Windows drive letters are only
supported in a single-server Vantage installation.
In a multiserver installation, all Telestream servers must join the same Microsoft
Windows domain as the EMC Isilon cluster. Please refer to the support center for
more information about authentication sources. Each Telestream Vantage server must
use Microsoft Windows Active Directory user authentication for the installed Vantage
services.
Figure 2. Vantage services
On each Vantage server go to Programs, Administrative Tools, and launch the
Services console. Scroll to the services beginning with the prefix Vantage, right click,
select properties, and select the “Log On” tab. By default, the Vantage services will
install configured to run as “Local System.” Change the Log On service property for
each Vantage service to use a dedicated Microsoft Windows Active Directory domain
user account. Do not use a domain user account for the Vantage service authentication
that is already in use by an employee or another application. If the account is subject
to a “lock out” policy after too many domain authentication failures, the entire array
could go offline until the account is unlocked. The Active Directory user account used
for Vantage services should have the “User cannot change password” and “Password
never expires” account options enabled.
Telestream Vantage management
In order to simplify Vantage workflow creation and management, determine which
storage locations are required and create a Vantage Store or Vantage Folder for
each UNC location under the storage section of the Vantage Management Console.
A Vantage Store is a managed location that will delete files once they are no longer
referenced by a Vantage workflow binder. Files on OneFS directories configured as
Vantage Stores are deleted upon job completion or catalog expiration date. Since
some transcode workflows create temporary media files that are cleaned up at the
Best practices for integrating Telestream Vantage with EMC Isilon OneFS
8
completion of the job, Vantage Stores can be particularly helpful in managing the
proliferation of temporary files that are not deleted due to an unexpected job failure.
Figure 3. Vantage Management Console
A Vantage Folder is an unmanaged file location that does not delete files. Vantage
Folders may be configured in either the Vantage Console or in the Vantage Workflow
Designer. Vantage Folders are managed in an address book for easy access when
building new workflows in Vantage Workflow Designer.
Vantage workflows are initiated via an API call including the UNC path to the source
file for the job or a “monitor” action defined in the Vantage Workflow Designer. Vantage
monitor actions include watch actions and associative actions. A watch action will poll
an EMC Isilon SMB shared folder or FTP site to check for a file pattern match, while an
associate action expects a specific file, such as a metadata file output by another
Vantage action.
Vantage watch actions are assigned to Vantage Stores and Vantage Folders in order
to assign a file matching pattern for an automation “watch folder.” The file matching
pattern should contain inclusive wildcards that identify a range of file extensions suitable
for the workflow and a filter to exclude unwanted client OS index files such as .DS_Store
and Thumbs.db.
Best practices for integrating Telestream Vantage with EMC Isilon OneFS
9
Figure 4. Vantage watch action configuration
Mac OS clients prior to 10.7 can also create “AppleDouble” structured data pointer
files for each file they write to an SMB volume. For example, a Mac OS 10.6 client
copying a 100 gigabyte file named media.mov to an SMB share can inadvertently
create a 100 gigabyte media.mov file and a 6 kilobyte AppleDouble file named
._media.mov. Apple support knowledge base article HT4017 details how to resolve
this issue by enabling Alternate Data Streams support on managed Mac OS clients.
In an environment containing unmanaged Mac OS clients, AppleDouble files are
hidden from Telestream Vantage watch actions by enabling the “Hide dot files”
advanced security parameter for the SMB share. Advanced SMB share parameters
are to be modified from the EMC Isilon Web administration user interface page for an
individual share, not the global SMB parameter UI. Contact your EMC Isilon support
representative before making uninformed changes to the advanced SMB parameters.
Telestream Vantage workflow automation is a powerful tool for today’s constantly
changing media content creation landscape. The constant addition and expiration of
workflows creates a reporting challenge for traditional media asset management and
traffic systems. When planning your workflow monitoring and reporting procedures,
talk to your EMC Isilon sales representative about EMC Isilon InsightIQ™. InsightIQ
maintains historical data about performance, capacity, and file analytics. Start your
InsightIQ data collection when you launch your first Telestream Vantage workflow.
The InsightIQ reporting interface is an invaluable resource for trending the storage,
bandwidth, and file system impact of your workflows. InsightIQ allows media
professionals to intelligently budget for the storage and network bandwidth
their workflows will need, before it is needed.
Best practices for integrating Telestream Vantage with EMC Isilon OneFS
10
About EMC Isilon
As the global leader in scale-out storage, Isilon delivers powerful yet simple solutions
for enterprises that want to manage their data, not their storage. Isilon products are
simple to install, manage and scale, at any size. And, unlike traditional enterprise storage,
Isilon stays simple no matter how much storage is added, how much performance is
required or how business needs change in the future. Information about Isilon can be
found at www.EMC.com/Isilon.
Best practices for integrating Telestream Vantage with EMC Isilon OneFS
11