Technical Case Study
This Is What Agility Looks Like
How Revlon IT harnessed big data with a private cloud
and global master data model
By David Giambruno,
Senior Vice President and CIO;
Ben Gent, Global Storage Engineer;
and Michael Cannella, Data and
Global Network Engineer
If you visit Revlon, Inc.’s data center in Oxford, North Carolina,
don’t blink, or you’ll miss the infrastructure. Just two racks
house roughly 3.6PB and 800 virtual servers that process
an average of 14,000 transactions per second (TPS) from
systems around the world, with 99.9999% uptime. When
people walk into our data center, they ask, “That’s it?”
The answer is yes, and it runs everything.
The picture was starkly different just a few years ago, when business intelligence
was buried in 21 separate enterprise resource planning (ERP) systems. High
management overhead kept our IT team from saying yes to requests for new
applications. Buildings didn’t even have Wi-Fi access, so chemists had to
jog back and forth between mixing vats and manufacturing control systems
to adjust product formulations.
This case study describes how we transformed Revlon IT by building a private
cloud and putting in place a global master data structure. Along the way, we
converted IT from an impediment to an enabler for agile strategic planning,
product development, manufacturing, warehousing, distribution, and operations.
Now systems work for people instead of people working for systems.
We accomplished the transformation in three phases: building a private cloud,
insourcing disaster recovery (DR), and building a global master data model.
Phase 1: Building a Private Cloud
We began the transformation effort by bulldozing the existing IT environment.
This included re-IPing all devices and systems and building a private cloud.
Our guiding principle for the cloud architecture was simplicity: single vendors
for storage, servers, virtualization, and networking and a single Domain Name
System/Dynamic Host Configuration Protocol (DNS/DHCP) server, server image,
and desktop image.
Revlon Data Center
Infrastructure Today
• Simpler, smarter data management:
– 120% production storage utilization
– 1,500% disaster recovery storage
utilization
– 1 person to manage 18 global
NetApp® storage systems
• Flexible data scaling:
– 5 times more storage capacity
at the same cost
– 14,000 transactions per second
• Highly available and reliable data:
– 99.9999% (six nines) availability,
with fewer than 13 seconds of
downtime annually
– 15,000 application moves monthly
– 30TB of changes weekly flowing over
the network to the DR data center
• Consolidation cost reduction:
– 56% server reduction, from 160 to 70
– More than $70 million in cost savings
and cost avoidance in past two years
– 72% decrease in data center power
consumption
•Agility:
– 70% faster application deployment
• Efficient project completion:
– 425% increase in the number of
IT projects delivered
We selected NetApp as our storage and data management vendor, deploying
NetApp FAS6000 and FAS3000 systems in the main data center and NetApp
FAS2000 systems in four remote facilities around the world. Had we gone with
a multivendor storage environment instead, we would have had to invest
resources in making them play well together. As it is, we can spend that
time developing new business intelligence applications.
After deploying the production infrastructure, we virtualized servers and desktops,
beginning with low-risk applications such as print and file servers, moving on
to departmental applications, and finishing up with mission-critical applications
such as ERP and manufacturing resource planning (MRP). Today, more than
530 applications—97% of the total—and all desktops are files (virtual machines)
running on VMware vSphere® and hosted on high-availability NetApp FAS6040
and FAS6240 systems. We’re running at 99.9999% uptime—fewer than
13 ­seconds of downtime annually. The private cloud processes 15,000
application moves monthly.
The NetApp systems also host SAP® front-end software; electronic data
interchange (EDI) transactions; Microsoft® SharePoint® Portal Server; the
mail transfer agent (MTA) used by IBM Lotus Domino; DNS/DHCP servers;
time tracking; applications based on UNIX®, including SAP databases and
bolt‑ons; Oracle® E-Business Suite Financials; and dozens of other businesscritical Oracle applications such as the Revlon manufacturing support and
legal systems.
Server virtualization enabled us to consolidate from 160 to 70 servers, a
56% reduction. The smaller footprint reduced our global power consumption
by 72%.
Redundancy doesn’t matter in our environment. We have only one server
connection because if we lose a server, applications move to another server
in the cloud.
– 70% reduction in the time to deliver
IT projects
– 99.6% of projects completed on
budget and on time
Timeline to IT Agility: 0-100% in Five Years
No incremental spending to achieve results.
Infrastructure
Simplification
Global Cloud
Deployment
18 months
Global Cloud
in Production
3 years
5 years
• Reduced costs with
$70+ million in savings
and avoidance
• 425% more IT projects
• Increased business
capabilities
2
Phase 2: Insourced Disaster Recovery
Next, we increased the return on investment from NetApp technologies
by bringing disaster recovery in house, an effort that paid for itself in just
seven months.
Because servers are now files, we can quickly replicate every production server
to a NetApp FAS3070 system located at our DR site, using NetApp SnapMirror ®
replication technology. We transmit 30TB of changes weekly to the DR data center.
David Giambruno
Senior Vice President and CIO
Revlon
David Giambruno joined Revlon as
CIO in 2006. He is responsible for
driving global technology capabilities
to provide competitive advantages
to the company’s business units
and shareholders. The breadth and
strategic nature of this project and the
speed at which it was accomplished
demonstrate his technical and business
leadership. He and his team have
transformed Revlon’s IT operations.
In our four global factories, data centers have been replaced with preloaded
“mini-me” data centers, which are half racks populated by multiple NetApp
FAS2020 and FAS2050 systems and a commodity switch and server. We like
to say we went from two racks of mess to one-half rack with 10 times the
performance at one-eighth the cost. Like the production data center, the mini-me
data centers replicate data to the DR site using NetApp SnapMirror software.
We test disaster recovery quarterly, bringing up an instance of all of Revlon—or
an individual country, site, or application—in the DR data center. The solution
was put to the test in 2011, when a fire destroyed most of the Venezuela facility.
With our previous infrastructure, recovery would have cost millions of dollars
and taken one to two weeks while people flew back and forth with suitcases full
of tape backups. With the NetApp DR solution, in just one hour and forty-five
minutes we brought up factory, EDI, and ordering systems in the DR data center
and spun up virtual desktops for employees so that they could work from home
or business partner offices the next day. We would have been back in business
even sooner if our storage engineer had not been on vacation with his phone
on silent.
We took the same steps to make sure Hurricane Sandy didn’t disrupt the business.
As Sandy came barreling toward the Metro New York area in November 2012,
we used NetApp SnapMirror replication technology to move our New York and
New Jersey data center operations to the North Carolina data center—in one
hour. After we confirmed the facilities had escaped damage, we moved the files
back over the network to their original data centers.
Phase 3: Corralling Big Data with Master Data Model
After building the private cloud and DR infrastructure, we turned our attention
to transforming big data—3.6PB worth—from a burden to a business asset.
Revlon tracks 660 million SKU attributes monthly, and our job in IT is to convert
this data to actionable information. Previously, requests for new business
intelligence applications had to wait until we added storage capacity; created
a mirrored copy of production databases; and performed extract, transform,
load (ETL) operations—not exactly a recipe for agility.
We came up with a unique solution to the big data challenge by creating a
global master data structure that automatically sorts, organizes, and structures
all Revlon data originating anywhere in the world. Every 15 minutes, Revlon’s
infrastructure pulls new data—including structured data as well as unstructured
data—from all global systems. Microsoft Data Quality Services (DQS) cleanse
and profile the data, normalizing it without the overhead of ETL.
After designing the master data model, we created more than 20 multiterabyte
parent datasets. Now development and test engineers use NetApp FlexClone®
technology to clone the datasets in an instant and run analysis in hours instead
of weeks. The NetApp Data ONTAP® operating system creates these clones with
zero storage overhead. Only new writes take up storage space, eliminating the
costs ordinarily associated with database copies. After using clones, developers
can either make them permanent or destroy them.
3
The ease of creating zero-overhead clones is transformative for Revlon.
There is no cost to mine the data and experiment. Instead, the NetApp
systems are simply providing added value while they spin platters. We’ve used
clones for large-scale application testing, change control, creating golden
images, and even modeling the effect of different e-mail retention policies
on storage requirements.
Ben Gent
Global Storage Engineer
Revlon
Ben Gent has worked for Revlon
since 2001. He held positions with
the help desk and PC support before
stepping into his role as global storage
engineer, in which he is responsible
for managing 3.6PB of data with
30TB of changes per week—a job
that previously required 10 people.
We think about big data in a different way than most companies. The traditional
model has applications at the center, surrounded by big data that the IT team needs
to ETL. In our model, big data sits in the center, with applications surrounding
it (Figure 1).
Current Big Data Practices
Revlon Big Data Practices
Application
Portfolio
Figure 1) Revlon big data practices.
Agility Means Saying Yes to the Business
The combination of the NetApp infrastructure, our global master data model,
and mobility solutions lets us wield technology as a competitive advantage.
We like to say we’ve taken the infrastructure out of the way so that the business
doesn’t even notice it anymore. We can deliver our application portfolio to any
device, anywhere in the world. And we’ve accelerated new application delivery
by 70% on average.
The proof of this newfound agility is that Revlon IT delivered 425% more
projects in 2012 than in 2007, reduced time to deliver a project by 70%,
and completed 99.6% of projects on budget and on time. That’s our key statistic,
and it’s largely the result of time savings from NetApp’s service automation and
analytics tools. For example, waiting time to receive a server has dropped from
6 to 8 weeks to 15 minutes. Our business users never need to wait for servers,
connectivity, or storage. If a department wants to try a new application, we
generally have it up and ready before the meeting is over, and it costs us nothing.
The new agility ripples through Revlon:
• Supply chain optimization. The first application to take advantage of the
master global data model was a global inventory report with drill-down capabilities, which distribution teams can access from tablets and smartphones.
Around 80% of Revlon users just want key performance indicators (KPIs)—
actionable information they need to do their job. Now they don’t need to
take the time to log into SAP or Oracle for commonly needed metrics.
• Increased manufacturing efficiency. Instead of climbing up and down ladders
to mixing tanks to check readings on wall-mounted computers, Revlon chemists
use iPad® devices with custom apps to adjust settings on a formulation system
hosted on NetApp storage. Not having to walk back to the office to adjust a
formula avoids materials waste and manufacturing delays.
4
• Automated picking and packing. Rather than packing shipments based
on a paper report that might be out of date, warehouse employees refer to
a screen in their truck to see where to go, what to pick, and where to place
the merchandise. Cutting down the time to package a product and get it out
of the building increases the rate of inventory turnover. And when the workers
in the truck can make accurate moves, we avoid vendor-imposed financial
penalties for incorrect shipments.
Michael Cannella
Data and Global Network Engineer
Revlon
Michael Cannella has been a data
and global network engineer with
Revlon since 2007. During his tenure,
he has contributed to the design and
implementation of the Revlon global
private cloud and virtualization
platform, an infrastructure that has
transformed IT and helped align
it to Revlon’s business.
• Mobility. The NetApp infrastructure is even helping to improve work-life
­balance. For example, finance teams working long hours during month-end
close activities can access their virtual desktops on NetApp storage from
home, using an iPad. And our IT team can change a server configuration
or move data during the week instead of coming in on the weekend.
• Reinvented help desk. The IT team processed 239% more help desk tickets
monthly in 2012 than we did in 2007, a benefit of spending less time on storage
management. We resolve 85% of issues on the first call, a rare achievement.
We’ve also seen a huge drop in the number of help desk tickets for problems
and a correspondingly huge increase in requests from people wanting to get
work done in a more efficient way.
In summary, the NetApp infrastructure and our master data model have transformed big data from a burden to a competitive advantage, even while cutting
costs. Overall, IT has improved the bottom line for the business in the past
two years through more than $70 million in cost savings and cost avoidance.
In addition, we’ve practically eliminated the incremental costs and wait time
to experiment and are limited only by our imagination.
Revlon employees, too, enjoy a new relationship with data. From anywhere,
any time, they can connect to the information they need to do their jobs more
efficiently and effectively.
How We Did It
NetApp storage infrastructure allowed us to disconnect innovation from incremental costs for networking, servers, storage, and staff time. Reasons include:
• We can commit resources without having a physical disk, a benefit of
NetApp thin provisioning. Utilization rates for production and DR storage
are 120% and 1,500%, respectively.
• Automated processes keep staffing requirements down. For example, NetApp
OnCommand ® Unified Manager provides a single view of storage management,
provisioning, and protection. NetApp Workflow Automation creates a policybased data infrastructure that we use to construct or edit workflows. We use
NetApp OnCommand Insight software to plan capacity requirements and
make sure that availability and performance meet service-level expectations.
Before, we needed 10 people to manage storage. Today, people are amazed
to hear that just one person manages 18 global storage systems with thousands
of spindles.
• We lowered storage costs using NetApp Flash Pool™, which is part of NetApp
Data ONTAP. Flash Pool lets us mix solid-state disks (SSD) and relatively
inexpensive SATA drives within a single aggregate, or “hybrid aggregate.”
Adding SSDs to the aggregate increased the performance of our busy SATA
disks by a factor of ten, providing performance similar to serial-attached SCSI
(SAS) disks at far less cost. We have been able to achieve a similar experience
with NetApp Flash Cache™. We were able to defer buying new hardware until a
budget cycle could be built and engineering work done to find the best solution.
Configuration is basically zero effort. As of this writing, we’re also completing
testing of NetApp Flash Accel™, which lets us use relatively inexpensive SATA
disks by minimizing latency by up to 50%. This gives our scientists and
marketers faster access to the data that drives product creation and sales.
5
Our business users can continue to access data even during system maintenance
because of the nonstop operations capability in the NetApp Data ONTAP
­operating system. We no longer need to schedule downtime for maintenance
and caching because NetApp SnapMirror replication technology can move
any virtual server, including ERP and other critical applications, between data
centers. When we upgrade our server hardware, we can move virtual machines
to the DR site while they continue to run and then move them back when the
server upgrade is complete. Case in point: In 2012, we replaced Revlon’s
global SAN core and data center server with no outages.
Another way we protect data is by using NetApp embedded security technologies,
such as NetApp Storage Encryption, role-based access control (RBAC), and
embedded antivirus.
Project Phases
PROGRAM
STEPS
Built private cloud in internal data center
• Built virtual server platform using NetApp
storage systems
• Deployed SAN
• Implemented wide area file services
Built global disaster recovery site
• Replicated all sites to internal DR site in
United States using NetApp SnapMirror
replication technology
• Automated backup and recovery, 7TB weekly
in just one building
• Tested quarterly
Shipped “mini-me” data centers
(half racks containing NetApp FAS2000
series storage systems) to regional
data centers
• Migrated application and desktop files remotely
• Replicated data to DR site
Virtualized all resources: application
servers, domain controllers, databases,
file servers, and print servers
• Replicated everything rather than trying to
understand all dependencies to selectively
replicate
• Standardized images
What’s Next
We’re continually looking for ways to increase the value of our NetApp investment
to the business, and some plans include:
• Simplifying the application portfolio: Our goal is to shrink the portfolio by 78%
by 2014. As part of that effort we’ll collapse 21 ERP systems into one.
• Delivering marketplace applications that employees can download and use
on iPhone® or iPad devices.
• Extending cloud access to partners and suppliers, giving them their
own applications.
• Implementing an active-active architecture.
• Deploying NetApp Flash Accel to production environment, accelerating
application performance so that product development and marketing
teams can work faster and increase Revlon’s competitive edge.
6
About Revlon, Inc.
Revlon is a global color cosmetics, hair color, beauty tools, fragrances, skincare,
anti-perspirant deodorants and beauty care products company whose vision is
Glamour, Excitement and Innovation through high-quality products at affordable
prices. For more information, please visit www.revlon.com.
About NetApp
NetApp creates innovative storage and data management solutions that
deliver outstanding cost efficiency and accelerate business breakthroughs.
Discover our passion for helping companies around the world go further,
faster at www.netapp.com.
Go further, faster
®
NetApp Products
• NetApp FAS6000, FAS3000,
and FAS2000 storage systems
• NetApp Data ONTAP
operating system
• NetApp Flash Cache
• NetApp Flash Pool
• NetApp FlexClone technology
• NetApp Operations Manager
• NetApp OnCommand
management software
• NetApp SnapManager ®
for Oracle
• NetApp SnapManager for
SQL Server ®
• NetApp SnapManager for
Virtual Infrastructure
• NetApp FlexVol ® technology
• NetApp deduplication
• NetApp SnapMirror replication
technology
• NetApp thin provisioning
Protocols
• NetApp SAN (FC and iSCSI)
with CIFS and NFS
Third-Party Products
•VMware® View ® 5.1,
VMware vCloud Director ®,
VMware vSphere, and
Site Recovery Manager
• IBM Lotus Domino mail
• Oracle E-Business Suite
Financials
• SAP database
• Microsoft Exchange 2010
and SharePoint 2010
• Juniper switches
Another NetApp
solution delivered by:
www.netapp.com
© 2013 NetApp, Inc. All rights reserved. No portions of this document
may be reproduced without prior written consent of NetApp, Inc.
Specifications are subject to change without notice. NetApp, the
NetApp logo, Go further, faster, Data ONTAP, Flash Accel, Flash Cache,
Flash Pool, FlexClone, FlexVol, OnCommand, SnapManager, and
SnapMirror are trademarks or registered trademarks of NetApp, Inc.
in the United States and/or other countries. iPad and iPhone are
registered trademarks of Apple Inc. Microsoft, SharePoint, and SQL
Server are registered trademarks of Microsoft Corporation. Oracle
is a registered trademark of Oracle Corporation. SAP is a registered
trademark of SAP AG. View, VMware, VMware vCloud Director, and
VMware vSphere are registered trademarks of VMware, Inc. UNIX is a
registered trademark of The Open Group. All other brands or products
are trademarks or registered trademarks of their respective holders and
should be treated as such. NA-168-0213
Follow us on: