Server Virtualization Assessment
– Tools and Techniques
Chicago ISACA Chapter 8/11/2011
Michael Hoesing CISA,
CISSP, CCP, ACDA, CIA, CFSA, CMA, CPA
mhoesing@mail.unomaha.edu
Anything discussed herein should be tested thoroughly in a lab
environment before use in production. Opinions are those of the author
and not conference sponsors, employers, clients, past, present or future.
Don’t sue me; I have no money.
www.isaca.org
Slide 1 of 41
Server Virtualization Assessment Objectives
• Virtualization Definitions, Background, Scope
• Risks and Controls
• Assessment Approaches and Tools:
• Assessment Examples
– VM (Guest) Sprawl
– ESX Console Operating System
(COS)Configuration
• Notes for vSphere 5
Slide 2 of 41
Background, Scope
Slide 3 of 41
BACKGROUND
• 2004 ++ Virtualization Spreads
• 2007 Gartner declares virtualization security
important
• 2007 to Today risk and security/control
techniques and products related to
virtualization evolve
• Now , and before now, we should evaluate how
effective are those security techniques and
controls (assessment)
• Business can’t live without speed to
deployment
Slide 4 of 41
SCOPE
• Virtualization Scope – ESX servers hosting
•
•
•
guests
Not Included – (only so much can be done in
1.5 hours) VDI, Hyper-V, Xen (Citrix & other
variants)
Some risk topics reach beyond ESX (policy,
process, procedure) if you are going to secure
an ESX environment you must think beyond the
COS
Some topics should be in scope but their
complexity is best covered separately (storage,
backups)
Slide 5 of 41
Risks & Controls
Slide 6 of 41
RISKS & CONTROLS – a list of 10
1. VM/Guest Sprawl
• Policies, Procedures,
Inventory Practices,
Reporting, Assessment
• Standards,
Monitoring,
Assessment
2. Host MisConfiguration
3. Network
Segmentation
4. Remote Access
Slide 7 of 41
• Deploy Segregated
Management,
Production and IP
Storage Networks
• SSH , SSL, access &
account controls
RISKS & CONTROLS – a list of 10
(cont)
5. User Account Access
& Roles
6. Single Point of Failure
• Policies, Procedures,
Least Privilege
7. Integration
• Strategic Architecture,
Capacity Planning
• Training
• Physical Security
8.
9.
10.
11.
12.
Staff Skills
Architecture (Blue Pill)
Software Licensing
I lied # 11 Appliances
#12 Guest Escape
VMSA-2009-0006
Slide 8 of 41
• Backups, Continuity
Planning
• Policy, Monitoring
• QA, Certification
Processes, Vendor Mgmt
• Patch Process
Assessment Approaches and Tools
Slide 9 of 41
ASSESSMENT APPROACH
• The Approach - 1.) a standard 2.) gather metrics
•
3.)compare metrics to the standard and cite
variances
Standard –
• a.) yours, if you have created a document, congratulations
• b.) VMware Hardening Guide(s)
http://www.vmware.com/files/pdf/vi35_security_hardening_wp.pd
f http://communities.vmware.com/docs/DOC-15413
• c.) CIS – ESX 3.0, 3.5, 4.x Xen http://cisecurity.org/benchmarks.html (also
has an XCCDF assessment tool (CIS-CAT) for members for 3.5 and 4.x)
• d.) DISA STIG
http://iase.disa.mil/stigs/stig/esx_server_stig_v1r1_final.pdf
Slide 10 of 41
ASSESSMENT APPROACH (cont)
• More Standards –
• e.) NIST 800-125 Jan 2011
http://csrc.nist.gov/publications/nistpubs/800125/SP800-125-final.pdf
• f.) PCI/DSS June 2011
https://www.pcisecuritystandards.org/documents/Virt
ualization_InfoSupp_v2.pdf
• g.) NSA http://www.nsa.gov/ia/_files/support/I733009R-2008.pdf
• h.) Vendors (HyTrust), consultants, books (Ed
Haletky, Scott Lowe, Siebert…)
Slide 11 of 41
ASSESSMENT APPROACH – Audit
Programs
• ISACA –
•
•
Whitepaper – issued Oct 2010 risks, audit approaches
http://www.isaca.org/KnowledgeCenter/Research/ResearchDeliverables/Pages/Virtualizat
ion-Benefits-and-Challenges.aspx
Audit program issued Jan 2011, GRC level
http://www.isaca.org/KnowledgeCenter/Research/ResearchDeliverables/Pages/VMwareServer-Virtualization-Audit-Assurance-Program.aspx
SANS talk through
http://www.sans.org/reading_room/analysts_program/VM
ware_ITAudit_Sep09.pdf
Mine (come to the hands-on class), mix of
process/procedure and detailed metrics
Slide 12 of 41
GATHERING METRICS – SOME
THOUGHTS
• In 50 mins all I can do is name-drop, you do the
research in your environment/strategy/risk appetite
• Not a Bake-off, not a Best-of, I can only relate what
worked in my lab (see bullet one above), any product
not mentioned just means I have not installed it yet
• Good News, lots of products to chose from , list
grows almost daily (bad news, that expands due
diligence time)
• Some tools Work in a Virtual Appliance, some tools
have both a physical and virtual appliance
• Key – does ingress and egress to/from the Guest
allow the product to do its Job (patching, AV, config
assessment)
Slide 13 of 41
GATHERING METRICS – SOME
THOUGHTS (cont)
• Free Tools – great price, don’t scale well
• Some tools inventory the Virtual Center
database, some tools enumerate raw data
• No one tool does everything, run multiple tools
for corroboration and completeness
• Tools that use a RHEL baseline, take care in
reviewing, but maybe 80-90% correct
• In a lab, build an ESX server (and vCenter) with
the vendor defaults, and build a second ESX
server with your organization’s standard build,
for education purposes and to calibrate tools
Slide 14 of 41
METRIC GATHERING TOOLS
• Interviewing and Document Review for policies,
standards, procedures, training
• !Free! Tools –
• console CLI, and vSphere remote CLI
• CIS-CAT 2.2.7 June/30/2011 (for members)
ESX 3.5 and 4.x benchmark XCCDF test
scripts
• VIToolkit & Powershell, (now called vSphere
PowerCLI 4.1 U1)
• esxcfg-xxx commands various
(i.e. esxcfg-firewall –q)
• esxcfg-info – dump of everything, load into ACL
and search
Slide 15 of 41
METRIC GATHERING TOOLS (cont)
• More Free Tools:
• vmware-vim-cmd hostsvc/ = grep /net/info or
grep /storage/info (careful, many of these
commands change settings, stick with the ones
with the word ‘info’)
• Configuresoft (Ionix) ComplianceChecker,
Tripwire configcheck, (ESX 3)
• From VMware - VI API, VIX API (allows files xfer
from guest) , Perl API, CIM API (risks of rolling
your own = script storage security, stored
passwords, change management, version
management)
Slide 16 of 41
METRIC GATHERING TOOLS (cont)
• More Free Tools:
• Bastille – remember to run in the –assess mode,
not the harden mode (3.0.9-1.0)
• DISA – SRR (security readiness review evaluation
script) watch these, they may harden if not run
correctly
• LSAT – works on 3.5 and before, but the MD5
process will try to analyze the very large vmdk
disk files, this is time consuming and could crash
running guests (note : does not work in vSphere, C
compiler is removed)
Slide 17 of 41
METRIC GATHERING TOOLS (cont)
• Existing Management Tools - (vCenter, Update
Mgr, Lifecycle Mgr, Veeam & others)
• Security Tools (Reflex, Catbird, BlueLane &
others)
• Commercial Tools – (Configuresoft [Ionix], Ecora,
Tripwire, & others)
• Hy-Trust - won a bunch at VMworld 2009,
access control and enhanced logging
• Vkernel Optimization Pack – inventories, finds
underutilization
• VMsafe vendor tools, Host Profiles(if using
Enterprise Plus)
Slide 18 of 41
Assessment Examples
VM (Guest) Sprawl
Slide 19 of 41
SPRAWL - CLI
• Free Tools – Command Line Interface (CLI)
ls –lR /vmfs/volumes/* | grep vmx
• Or the ‘find’ command (does not follow sym
links)
•
•
•
•
•
•
•
•
•
•
•
-rwxrwxrwx 1 root root 4831838208 Jul 7 2007 BLVS-flat.vmdk
-rwxrwxrwx 1 root root
331 Jul 7 2007 BLVS.vmdk
-rwxrwxrwx 1 root root 8589934592 Jul 7 2007 BLVSMgr-flat.vmdk
-rwxrwxrwx 1 root root
336 Jul 7 2007 BLVSMgr.vmdk
-rw------- 1 root root 872415232 Sep 23 10:10 Reflex-VSA-Template-flat.vmdk
-rw------- 1 root root
480 Sep 23 10:10 Reflex-VSA-Template.vmdk
-rw------- 1 root root 4294967296 Oct 8 11:37 Reflex-vsc-flat.vmdk
-rw------- 1 root root
499 Oct 8 00:50 Reflex-vsc.vmdk
-rw------- 1 root root 6442450944 Sep 29 01:59 RHEL-4-4-ES-flat.vmdk
-rw------- 1 root root
339 Sep 29 01:57 RHEL-4-4-ES.vmdk
-rw------- 1 root root 16791552 Mar 17 2008 SLES10-SP1-000001-delta.vmdk
Slide 20 of 41
SPRAWL – CIS-CAT
Free Tools CIS-CAT (if a member) will list VM’s
with non-compliant vmx config files (not a
complete inventory but a good start on what
needs correction)
Slide 21 of 41
SPRAWL – PowerCLI 4.0
VI Tools for Windows & Powershell now
named vSphere PowerCLI 4.1 (partial
script)
•
•
•
•
•
•
•
•
•
$VC = Connect-VIServer 192.168.1.21 -User XXXXXX -Password
XXXXXX
$VMs = Get-VM | format-table -property name
$Datastores = Get-Datastore | Format-Table -property Name
$VMXlist = " "
$i = 1; while ($i -le $Datastores.length-4)
$Datastore = Read-Host "Enter Data Store Name, like storage1*
from the list above "
get-childitem -recurse -include *.vmx | format-table -property name
>> c:\vmxlist
$i +=1
Then compare the two files (VM list and vmx list) with diff, ACL, or
manually
Slide 22 of 41
SPRAWL – vCenter
• Existing Management Tools - Virtual Center
Slide 23 of 41
SPRAWL – Reflex
• Third Party Security Tools – Reflex
Slide 24 of 41
SPRAWL – Configuresoft (IONIX)
• Commercial Configuration Assessment Tools
– Configuresoft (Ionix)
Slide 25 of 41
SPRAWL – Ecora
• Commercial Assessment Tools – Ecora
Slide 26 of 41
SPRAWL – Honorable Mention
• Anything that Monitors Usually has an Inventory Component
•
Akorri Balance Point
BMC Performance Manager
•
CA ASM (Unicenter)
eG Innovations Enterprise
Suite
•
Embotics V-Commander
HP Operations Orchestration
•
IBM Tivoli Monitoring for Virtual Servers
•
ManageIQ EVM Suite
Netuitive SI for VMware
•
Quest vFoglight
Symantec Altiris
•
Tideway Foundation
Veeam Monitor
•
SPI for VMware
vmInformer
Slide 27 of 41
Assessment Examples
Host Configuration
Slide 28 of 41
HOST CONFIGURATION – CIS-CAT
Categories
• CIS-CAT 9 Categories (3.5)
Slide 29 of 41
HOST CONFIGURATION – CIS-CAT
Benchmark Items
• CIS-CAT 29 Benchmark Items (3.5)
Slide 30 of 41
HOST CONFIGURATION – CIS-CAT Detail
Assessment Test and Results
• 1.2.3 Recommended Boot services (3.5)
Slide 31 of 41
HOST CONFIGURATION – CIS-CAT
Categories
• CIS-CAT 12 Categories (4.1)
Slide 32 of 41
HOST CONFIGURATION – CIS-CAT
Benchmark Items
• CIS-CAT Benchmark 65 Items (4.x) (partial)
Slide 33 of 41
HOST CONFIGURATION – CIS-CAT Detail
Assessment Test and Results
• 9.1 Recommended Boot services (4.x)
Slide 34 of 41
HOST CONFIGURATION – Tripwire
• Commercial Assessment Tools – Tripwire
Slide 35 of 41
vSphere 5
Slide 36 of 41
vSphere 5
•
•
•
•
Released July 2011
Memory based pricing is new, and not popular
ESX COS is gone, ESXi the only choice
ESXi has hypervisor and console all on the same
partition, faster (vendor says)
• ESXi 5 has a firewall (iptables) ESXi 1-4 did not
• No (if configured as suggested) console access,
all access is remote
• Use vMA, remote CLI, and PowerCLI for audit
metric gathering or vCenter
Slide 37 of 41
vSphere 5 (cont)
• TPM (Trusted Processing Module) recognition
available (Intel’s TXT or AMD’s SEM , soon)
• Hope they Fixed These in 5 (ESXi 4.1 issues)
 Logs removed upon reboot  root password not set
during installation
 Tech Support Mode (from console)
 Remote Tech Support Mode (SSH), accesses Single
User Mode (root without any password if not set at
default, even with password root SSH is enabled)
 Reset System Configuration – resets an empty root
password (watch iLO and iDRAC)
Slide 38 of 41
Conclusion
Slide 39 of 41
SUMMARY
• Virtualized Infrastructure is Important to the
Organization and worthy of secure configuration
and periodic assessment of that state
• Standards are available for a starting point to
create/edit your organization's policy
• Tools are available, in all price ranges, to gather
metrics from an ESX environment
• Get the tools, gather the metrics, compare to the
policy/standard, cite the differences, improve
your security posture
Slide 40 of 41
Q and A
– If the question comes to you later
mhoesing@mail.unomaha.edu
–?
–?
–?
–?
–?
–?
Slide 41 of 41