Virtualization, Cloud and Big
Data
IT Operational Analytics to Cost optimize the “Virtual
Everything” IT Universe
David Wagner – TeamQuest Advocate
TeamQuest and the TeamQuest logo are registered trademarks in the US, EU and elsewhere.
All other trademarks and service marks are the property of their respective owners.
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Background
• Presentation is an adaptation of recent TeamQuest and
Forrester Research Webinars:
– “Business AND IT Analytics: You need both to optimize Business”
• James Staten, Vice President and Principal Analyst, Forrester Research
• David Wagner, Advocate, TeamQuest
• September 12, 2013
– “Beyond Virtualization: Accelerating the Road to Software Defined
Computing”
• Richard Fichera, Vice President and Principal Analyst, Forrester Research
• David Wagner, Advocate, TeamQuest
• August 15, 2013
• Recordings of both Webinars available: www.teamquest.com
All Forrester content with permission and noting attribution
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Agenda
• What is a Software Defined Data Center?
– Definition, evolution, capabilities , approaches…
• Business Analytics with IT data
– Definition, evolution, capabilities, approaches…
• Cost Optimization:
– Or… “The corner of Business Analytics, IT Analytics and SDDC meet
the discipline of Performance Analysis and Capacity Management”
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Software Defined Data Center
Rich Fichera, VP and Principal Analyst, Forrester Research
TeamQuest and the TeamQuest logo are registered trademarks in the US, EU and elsewhere.
All other trademarks and service marks are the property of their respective owners.
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Software defined computing
• An integrated abstraction layer that presents computing
infrastructure as pools of virtual and physical resources,
allowing users to dynamically compose them into Services
– Rich Fichera: Forrester, September 2013
• Forrester, VMware, and others (e.g. Bernd Harzog, The
Virtualization Practice) call this the “Software Defined Data
Center”
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Enterprises will deploy their new
virtual servers in dynamic pools
“What percentage of your total x86 servers OS instances are in each of the following categories?”
Today (N=486)
In four years (N=476)
On physical servers not virtualized
42%
23%
32%
31%
On relatively static virtual servers
13%
On a dynamic private cloud pool of virtual servers
5%
4%
Traditional hosting offering
3%
Hosted private cloud IaaS
3%
4%
Outsourced service provider
Public cloud IaaS
27%
1%
3%
7%
In four years, enterprise respondents expect to
move about 20% of their current physical
servers into a private or public cloud
Base: North American and European IT decision makers at enterprise firms using physical x86 servers
Source: Forrsights Hardware Survey, Q3 2012
© 2013 Forrester
Inc. Corporation.
Reproduction All
Prohibited
Copyright
© 2012Research,
TeamQuest
Rights Reserved.
6
New data center technologies
focused on reducing manual
assembly & config
Base: 1048 IT hardware decision makers at enterprises in North America and Europe
Source: Forrsights Hardware Survey, Q3 2012
© 2013 Forrester
Inc. Corporation.
Reproduction All
Prohibited
Copyright
© 2012Research,
TeamQuest
Rights Reserved.
7
So, enterprises see the benefit of data
center abstraction
• Respond faster to business spikes
– USAA’s mobile app created 10x the transaction volume than it
expected
• Deploy and refresh new applications faster
– Retailers need to capture their share of mobile commerce as it grows
from $6 to $31b (2016)
• Reduce upfront capital outlays and system operating expense
• Optimize resources for systems of engagement
• This is not the end of the road
Forrester content
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Virtualization remains highest IT
infrastructure priority, but cloud gaining
“Which of the following initiatives are likely to be your firm’s/organization’s top hardware/IT infrastructure
priorities over the next 12 months?”
(Percentage of respondents who answered “critical” or “high” priority)
Consolidate IT infrastructure via server consolidation, data
center consolidation, or server virtualization
78%
81%
80%
79%
Maintain or implement broad use of server virtualization
as the standard server deployment model
79%
79%
80%
77%
63%
63%
60%
61%
Automate the management of virtualized servers to gain
flexibility and resiliency
Build an internal private cloud operated by IT (not a service
provider)
Use cloud infrastructure-as-a-service (IaaS) at a service
provider
29%
23%
18%
36%
46%
2012 (N=1,036)
2011 (N=1,240)
2010* (N=1,037)
2009* (N=1,020)
35%
31%
28%
Base: North American and European IT decision makers at enterprise firms with 1,000 or more employees
*Question wording in 2009 and 2010 referred to IaaS as “virtual-server-as-a-service”
Source: Forrsights Hardware Survey, Q3 2009, 2010, 2011, 2012
© 2013 Forrester
Inc. Corporation.
Reproduction All
Prohibited
Copyright
© 2012Research,
TeamQuest
Rights Reserved.
9
Architectural challenges for SDDC
• Inclusion of legacy or non-virtualized hardware resources?
• Interoperability of multiple vendors’ converged infrastructure
systems?
• Software-defined networking approach – SDN vs. OpenFlow
vs. Cisco
• Management remains a mystery
– Lack of real time data
– Lack of global optimization strategies
Forrester content
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Management is more than monitoring
• SDDC management will place a premium on new
management models
– Real time data collection
– Embedded Analytics
– Ability to span multiple data source domains intelligently
• Analytics will have different goals – efficiency, cost, root cause
analysis
• Goals include workload-centric optimization, global cost and
energy efficiency, global availability … very tough challenges
for management vendors
Forrester content
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Software defined data center Considerations
• The term is largely unheard of, but infrastructure leaders generally
understand or can guess what this implies
• More senior leaders think of consolidated data centers of the past
– But many are “wait and see”
– “Sounds nice, but how do we get there?”
• Skepticism that one company can do servers, storage and network well
– That impression might be different when speaking to established converged
infrastructure customers (UCS/VCE/FlexPod/Pure customers)
• Server virtualization is seen as much more advance than SDN or software
defined storage
• Buy-in on the concept will require cross-silo strategy and collaboration and
threaten established centers of power in the data center
Forrester content
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Business Analytics with IT Data
James Staten, VP and Principal Analyst, Forrester Research
TeamQuest and the TeamQuest logo are registered trademarks in the US, EU and elsewhere.
All other trademarks and service marks are the property of their respective owners.
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
“We don’t Leverage IT
Data because…”
•
•
•
•
•
It’s a mess
It isn’t standardized
Value isn’t intrinsically measurable
Systems can’t talk to each other
It isn’t easily presented or understood
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Pop quiz:
What is the business
more worried about?
Adjusting to what we
know is changing in our
market
© 2013 Forrester Research, Inc. Reproduction Prohibited
Adjusting to what we
don’t know
15
True business visibility
is limited
What they know:
› Sales growth
› Seasonality
› Customer feedback
© 2013 Forrester Research, Inc. Reproduction Prohibited
What they don’t:
› How well services are
›
›
performing
Why customer
experiences are poor
What IT knows
16
Intelligence—not data—will
be the difference between
success and failure.
$600B
1.2 yrs
2.7 Zettabytes
Source: Forrsights Strategy Spotlight: Business Intelligence And Big Data, Q4 2012
We don’t leverage IT
data because . . .
› It’s a mess
› Isn’t standardized
› Value isn’t intrinsically measurable
› Systems can’t talk to each other
› Isn’t easily presented & understood
© 2013 Forrester Research, Inc. Reproduction Prohibited
18
But the business values this over all else
Improve the use of data analytics to
improve business decision making
Source: Forrsights Business Decision-Makers Survey, Q4 2012; Base: 1,170 EMEA business decision-makers
© 2013 Forrester Research, Inc. Reproduction Prohibited
19
What IT data can tell the business
1.
When & where customer experience breaks down
2.
Where inefficiencies lie in key business processes
3.
When & where performance problems really affect the bottom
line
4.
When legal issues arise (and how to prevent them)
5.
Find root causes that result in customer dissatisfaction
6.
Find and see past false indicators
© 2013 Forrester Research, Inc. Reproduction Prohibited
20
The Most Popular Types Of IT Analytics, How They Work, And Their Value
December 2012 Forrester Report: “Turn Big Data Inward With IT Analytics”
Topological Relationships Help Map Cause-Effect Phenomena
December 2012 Forrester Report: “Turn Big Data Inward With IT Analytics”
Statistical Analytics Uncovers Performance Patterns
December 2012 Forrester Report: “Turn Big Data Inward With IT Analytics”
Configuration Analytics Spots Changes And Policy Violations
December 2012 Forrester Report: “Turn Big Data Inward With IT Analytics”
What you can do about it
1. Unify IT data
• Centralize the analysis – views by role
2. Map the business to IT systems
• Then you can map business efficiency
3. Show the business what you find
• Then let them at the tool for their own analysis
© 2013 Forrester Research, Inc. Reproduction Prohibited
25
Business AND IT Analytics
Using both to optimize the business in a “virtualized
everything” world
David Wagner - TeamQuest
TeamQuest and the TeamQuest logo are registered trademarks in the US, EU and elsewhere.
All other trademarks and service marks are the property of their respective owners.
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• “You have what the business wants”
• “Map to business to IT
systems, then you can
map business efficiency”
– James Staten (9/12/13)
• “The business already has what you NEED”
• “Map IT to the business, then you can map IT
efficiency”
– Dave Wagner (10/04/13)
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
“We don’t use business data
because…”
• Our existing tools don’t accept it
– ETL into acceptable format(s) is a lot of work, cost
– Manual efforts won’t scale
• We have no expertise in business data
– It isn’t easily presented or understood
– We’ve never had the time to deal with it
• What can we really DO with it?
– Where do we start?
– What is the value?
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Topics
• Software Defined Data Center
• Business Analytics and IT
– Revolutionary new business technologies ->
Data growth
– Analytics for business, fed by IT
• IT changes: Cloud, Virtual, SDDC…
• IT performance & capacity management
impact
– Challenges
– An analytic approach
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Cloud, Virtualization and SDDC -> Changing Demands
on Enterprise IT
Business Services/Applications
SDDC
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
•
•
•
•
•
•
Asset Management
Smart Devices
Virtualized Storage
Virtualized Servers
Virtualized NW
Etc….
Customer
•
•
•
•
•
•
•
Call Center Operations
POS Optimization
Demand Response
Campaign Management
Credit & Collections
Billling
Etc….
Increasing Planning and Optimization Challenges
Business Services/Applications
SDDC
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
•
•
•
•
•
•
Asset Management
Smart Devices
Virtualized Storage
Virtualized Servers
Virtualized NW
Etc….
Customer
•
•
•
•
•
•
•
Call Center Operations
POS Optimization
Demand Response
Campaign Management
Credit & Collections
Billling
Etc….
Requirement: Align IT performance/capacity and
business across technologies
Big Data Collection
Underlying IT
Infrastructure
Outage
Management
Customer
Operations
Distribution
Automation
Asset
Management
Services
Enterprise
Performance &
Capacity Mgt.
• Correlate business
process & IT
performance
• Insight into how
business process
changes impact IT
Applications
• Understand IT costs
by business
unit/process
Server/OS
• Insight into business
process performance
all the way down to
storage components
Network
Storage
Business Intelligence
Aligned Business and IT Intelligence
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
32
Needing answers…
“What is the health of
the infrastructure?”
“What’s the risk
assessment?”
“How efficiently
are we using the
infrastructure?”
“How many of those
do we have? Need?
When?”
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
“What were the
exceptions? Do we
Care?”
“What’s the capacity
forecast?”
“Changes compared
to the last reporting
period?”
Capacity management:
Today’s IT optimization tradition
• Analytics
– Monitoring technology performance utilization trends, statistics
– “Siloed” problem analysis
– Planning focus: optimizing server cost versus performance
• Extremely technology-centric – in isolation
– Servers and mainframes: physical and virtual
– Storage and network: physical and virtual
• Big value and return, but also a big effort
– Requires highly trained staff; domain expertise
– Requires building a central, long-term repository (CMIS or PMDB)
– Problems with scalability of staff and tools … Politics!
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
New capacity management goals
• Build on traditional value
– Optimize more resources
– Amplify
– Accelerate
• Increase business relevance
– Valuable predictive analytics in business and service context
– Optimize efficiency
• Virtualization and cloud  SDDC: scale everything
– Many-to-many interrelationships; capacity more critical
– Back end performance now more critical
– Front-end devices -> performance and capacity
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Infrastructure optimization
• Continuous DevOps, consolidation,
repurposing, re-hosting, etc.
–
–
–
–
Least used (servers, virtual servers, storage, etc.)
Most power (consumption, carbon footprint, etc.)
Most expensive (total OpEx)
By application, over time… and many more
• Financial optimization – not utilization
anymore!
– Under-used doesn’t necessarily mean wasteful!
– Over-used doesn’t necessarily mean lost revenue/profit
• Goal: eliminate true waste, ensure
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Infrastructure optimization data
sources (some)
• Asset database
– Server CapEx, OpEx and licensing costs
• Service catalog
– Mapping of applications to servers/virtual servers
• Power consumption and cost
– kWh per server
– cost per kWh over time, versus dynamic demand, etc.
• Resource utilization, events, etc…
– By server, virtual server and application, storage,
network…
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
A Sub-optimal Approach:
BI-type IT Data Warehousing
• Build a huge “data mart” (i.e. PMDB)
Complexity = (data ETL) x (# sources) x (maint.
effort) x
(SDDC variability/dynamism) x …
+ Compliance: Data duplication, privacy, audit,
etc…
+ Lock in”
= Very costly and time-consuming
• Apply general purpose BI analytics
– Not focused on IT Resource optimization,
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
A Better Solution
INTRODUCING A “BIG DATA”
TYPE ANALYTIC APPROACH
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
39
A “Big Data” type analytic approach
• Federate existing data into purpose-designed
performance/capacity process
– Technology data (e.g. server, network, storage, etc.)
– Service data (catalog, metrics, tickets, etc.)
– Business data (Business Analytics, KPIs, plans, TXNs,
etc.)
• Automate analytics across all data sources
– Flexible and adaptive to dynamic SDDC environments
– Raw (commodity) data -> actionable information for IT
• Latency & throughput, not simple utilization rates or availability
• Single-pane-of-glass management
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
A “Big Data” Analytic Approach
SOME EXAMPLES
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
41
Analytic examples: Multi-data types
• Business, service, asset and power
– Continuous financial optimization
• Service tickets and performance/capacity
– Risk management
• Spanning pure technology silo challenges
– Improve availability and speed MTTR
• Risk prediction – by latency/use
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Financial optimization report
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Risk: Service desk and capacity
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Virtualized Servers AND storage together
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Servers AND storage — together
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Servers AND storage - together
Huge
Value
for
Virtual/Cloud
Environments:
Cross the Chasm!
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Managing Performance that Matters
TEAMQUEST PERFORMANCE
INDICATOR
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
48
TeamQuest Performance Indicator:
Automating proactive latency analytics
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Automated, exception-based analytics
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Automated queuing RCA
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Latency holistic RCA:
Server, workload, VM *and* storage
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Results
• Large Insurance
– Multi-year transition to dynamic grid and virtualization
– Improved resource utilization by over 400%
– Saved $25M over 3 years – expense deferral while maintaining SLAs
• Large Finance
– Deferred massive, planned data center build out by 18 months and
counting ($20M+ to-date)
– Reduced service risks, lowered operational costs, expanded staff
scope/reach
• Cloud AAS Provider
– Sped MTTR for client-facing services by several fold
• Telecommunications
– Sped client-facing infrastructure MTTR, cost-reduction, right-sizing
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
53
Assuring Peak Performance
NEW! TEAMQUEST RISK
PREDICTOR
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
54
TeamQuest Risk Predictor
• Know how long you've got until response time will suffer
• Rules by TPI, utilization, percent change in response time
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
55
Exception-based latency prediction:
by workload, VM, system…
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
56
Auto-predicted queuing analytics:
By workload, VM, system…
• Continuous, Rolling Prediction
• Automated Re-baselining
• Complete Components of Response Time (latency)
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
57
Additional analytic considerations
• Federation maps virtually all sources… but not all
– Low effort
– Rapid time to value
• Any chosen analytics must handle
– Multiple, disparate sources
– Modularly, combinatorial
• You will want rule-based automation of analytics
– Flexible, easy to modify
– Only way to “human scale” in dynamic, large IT environments
• Pick “low hanging fruit” for first project(s)!
– Use existing data sources instead of boiling the ocean
– Build from your wins
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Summary and Recommended Steps
• Inventory IT processes and tools
– Data collection
– Data storage
– Capacity Analysis/Reporting
• Inventory business KPIs, factor BI analytic initiatives
• Identify “low hanging fruit”
–
–
–
–
By Service
By BU
By Platform
By Technology/silo
• Get started bringing it together, don’t wait
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
59
TeamQuest: Insight Gained
“I know what my
risks to
performance are
and will be!”
“I know how
efficient my IT
environment is
and can be!”
“I know what we
have, what we need,
and when we need
it!”
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
“I know the
performance of my
infrastructure!”
“I understand the
exceptions and their
importance!”
“I have a Capacity
Plan!”
“I know what has
changed and why
from the last period!”
TeamQuest: Assuring your Future!
• Peak Performance
• Ideal Capacity
• Optimized Costs
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
61
Questions?
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
62