The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter Four
Are We Ready to Patch Yet?
Authors:
Anne Stanton
President, Norwich Group
Susan Bradley
Microsoft Small Business Server MVP
Chapter 4 – Are we ready to patch yet?
T
here is one key point that we need to emphasize. Even though we try to replicate our
systems as best as we can, there will be times when we cannot perform an exact test to
find all the expected issues that may arise in connection with deploying a patch. Thus, it is
critical that you have backups and system images before you begin to roll out patches.
The following are a few of the available backup and imaging solutions for Windows and Linux
environments:
• Symantec Ghost
• Native NTbackup
• Acronis True Image
• UniTrends
• Veritas
• Arkeia
• Ultrabac
Remember evaluate your network from a risk analysis standpoint, even if you have a proper
backup or image of those stations you are patching. What systems are the most at risk for the
patch in question? Carefully review security bulletins and assign patch priority based on the role a
system plays in your network. Look again at patch guidelines such as Security bulletin 04-0111 for
the Sasser worm. Security bulletin 04-0112 was a comprehensive bulletin that patched several
security issues. It covered vulnerabilities that exposed edge devices to denial of service issues.
“Chatter” on the listserves indicated that systems directly attached to the Internet and at the edge
of a network needed immediate patching. For this reason, some firms, under certain
circumstances, take the calculated risk of patching edge systems without pre-testing.
Patch “at risk” machines first
Identify regions in your network that are most at risk and assign zones. This is similar to the
“Organizational Units” concept in the Active Directory setup. Zone out and patch servers
separately from workstations, and build patch templates for each zone to scan and patch each
zone separately. Internet Explorer, Outlook Express, or Windows Media patches may be more
critical on desktops than they are for servers, as no one surfs or reads e-mail from server
machines. Furthermore, in Windows 2003, the Internet Explorer enhanced lockdown ensures that
most Internet Explorer patches are not highly critical. The reverse is true for desktops and
machines used by information workers who surf for research purposes. Here there is a higher
priority on Internet Explorer patches. Next, prepare templates for machines that you can patch
more quickly than other machines due to their role in the firm or because they do not require prepatch backup procedures. Develop zones so that you can patch different machines in different
ways.
For those devices that do not permanently attach to your network, you may need a combination
agent/agentless patch tool. We will address this in detail later, but the type of patch tool you have
may dictate how you process the patches. Whether you manually script it, use Software Update
Services, or use one of several available Patch Management software tools, your ability to control
the patch process dramatically increases the more you automate this process. Additionally
automation documents the change process and audits the installation of the patch process.
Even after you have properly tested, consider doing a controlled test roll out to designated
testers. This is particularly true for different types of patches such as a service pack that contains
many more code changes than a security patch does. After your test team has approved the
service patch for release, consider a staggered rollout. A testing environment can go only so far.
It can never truly anticipate all of the issues that may arise in a production network. In testing of
Windows XP SP2, we found that, even though we had no issues during testing, once deployment
began, we saw issues with desktops that had digital graphics cards installed. We needed to track
down an updated digital video driver. We had not deployed to the exact same hardware on which
we had tested and we encountered unforeseen issues during the rollout. Your pre-rollout testers
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 2 of 16
should be computer literate and capable of clearly reporting issues to your help desk team for
resolution. Once you begin patch rollout, notify employees so that they can be sensitive to the
pre- and post-patch issues they may see.
Screen shot of one available patch program, HFNetChkPro
Use patch tools to confirm compliance with policies
Use patch management tools to augment your firm’s policies and to audit the effectiveness of
your external vendors. Additionally you may have contractual lease agreements with external
vendors to manage systems. As part of these agreements, you can request that either you or
they monitor the status of patches.
Further, use your patch management best practices process to scan your systems periodically
between patch periods to ensure that users have not added vulnerable machines to your network.
Build this procedure into your agreements, and make it part of your normal control process so
that you maintain proper control of your company’s systems. Clarify polices regarding patch
responsibility. As we have stated earlier, if your vendors will not allow you to patch, you must
scan and monitor your system so you can isolate, protect, and defend.
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 3 of 16
Ecora Software graphical view of patch status
Capture feedback
Every organization has or should have formal or informal help desk capturing tools3 for tracking
issues relating to the patches you roll out. Ensuring that you quickly identify issues and work with
your vendors is extremely important. We have seen cases where a patch application changed the
way in which a third party program interacted with a system. When this happens, begin working
with the vendors. In the case of Microsoft patches, any issue caused by a security patch is a free
support4 call. Red Hat tracks issues in their bug database.5 It is also wise to build up tech support
relationships with antivirus vendors, firewall vendors, and line of business application vendors.
Determine if they have e-mail notification services for issues with security patches. With excellent
cooperation and communication, we can ensure the smoothest release process. Microsoft has
put in place processes to assist in testing their patches prior to release. As one industry observer
has noted, “Microsoft made good on the promise to beta test security patches with a limited
number of customers and partners. By testing the patches in real-world environments, this
program succeeds in increasing the quality of final patches.”6 A similar feedback process in your
environment contributes to patch process success.
Enforcement
We assume that your corporate policy mandates that all machines should have patches installed;
however, without enforcement of the policy, you systems remain exposed. Here is the rule:
“Policy First; Technology Second”. In the case of security policies regarding the level of patches
on systems, you can find guidance on setting and developing your policies on various web sites.
The SANS.org7 Security policy page highlights the following sites:
•
http://www.security.kirion.net/securitypolicy/
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 4 of 16
•
•
•
•
•
•
•
•
•
http://www.network-and-it-security-policies.com/
http://iatservices.missouri.edu/security/
http://www.utoronto.ca/security/policies.html
http://irm.cit.nih.gov/security/sec_policy.html
http://w3.arizona.edu/~security/pandp.htm
http://secinf.net/ipolicye.html
http://cio.berkeley.edu/policies.html
http://www.ruskwig.com/security_policies.htm
http://razor.bindview.com/publish/presentations/InfoCarePart2.html
There are also numerous examples of security policies on the SANS.org web page. The key
elements you can find from these documents include:
From the Server Security policy8:
•
Install the most recent security patches as soon as practical, the only exception being
when immediate application would interfere with business requirements.
From the Acquisition policy9:
•
•
Install a standard <Company Name> image on all hosts—servers, desktops, and laptops.
Business critical production servers that cannot be replaced or re-imaged must be
audited and a waiver granted by InfoSec.
From the Audit policy10:
•
Scanning period: <Company Name> and <Internal or External Audit Name> Scanning
Team shall identify in writing the allowable dates for scans to take place.
From the VPN connection policy11:
•
All computers connected to <Company Name> internal networks via VPN or any other
technology must use the most up-to-date anti-virus software that is the corporate
standard (provide URL to this software); this includes personal computers
Sample Test Lab Security policy
Given how critical it is to your test network, we provide an entire Test Lab Security policy
referenced and available from the SANS.org site. It points out the importance of separating the
Lab from the rest of the network. We highlight the test lab security policy in particular, as it is
imperative that you keep your test lab from infecting the rest of your production network. Isolate it
accordingly.
Internal Lab Security Policy12
1.0 Purpose
This policy establishes information security requirements for <Company Name>
labs to ensure that <Company Name> confidential information and technologies
are not compromised, and that production services and other <Company Name>
interests are protected from lab activities.
2.0 Scope
This policy applies to all internally connected labs, <Company Name> employees
and third parties who access <Company Name>'s labs. All existing and future
equipment, which fall under the scope of this policy, must be configured
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 5 of 16
according to the referenced documents. DMZ Labs and stand-alone, air-gapped
labs are exempt from this policy. DMZ labs must comply with the DMZ Lab
Security Policy.
3.1 Ownership Responsibilities
1. Lab owning organizations are responsible for assigning lab managers, a
point of contact (POC), and a back-up POC for each lab. Lab owners must
maintain up-to-date POC information with InfoSec and the Corporate
Enterprise Management Team. Lab managers or their backup must be
available around-the-clock for emergencies, otherwise actions will be taken
without their involvement.
2. Lab managers are responsible for the security of their labs and the lab's
impact on the corporate production network and any other networks. Lab
managers are responsible for adherence to this policy and associated
processes. Where policies and procedures are undefined lab managers must
do their best to safeguard <Company Name> from security vulnerabilities.
3. Lab managers are responsible for the lab's compliance with all <Company
Name> security policies. The following are particularly important: Password
Policy for networking devices and hosts, Wireless Security Policy, Anti-Virus
Policy, and physical security.
4. Lab managers are responsible for controlling lab access. Access to any
given lab will be granted by the lab manager or designee only to those
individuals with an immediate business need within the lab, either short-term
or as defined by their ongoing job function. This includes continually
monitoring the access list to ensure that those who no longer require access
to the lab have their access terminated.
5. The Network Support Organization must maintain a firewall device
between the corporate production network and all lab equipment.
6. The Network Support Organization and/or InfoSec reserve the right to
interrupt lab connections that affect the corporate production network
negatively or pose a security risk.
7. The Network Support Organization must record all lab IP addresses, which
are routed within <Company Name> networks, in Enterprise Address
Management database along with current contact information for that lab.
8. Any lab that wants to add an external connection must provide a diagram
and documentation to InfoSec with business justification, the equipment, and
the IP address space information. InfoSec will review for security concerns
and must approve before such connections are implemented.
9. All user passwords must comply with <Company Name>'s Password
Policy. In addition, individual user accounts on any lab device must be
deleted when no longer authorized within three (3) days. Group account
passwords on lab computers (UNIX, windows, etc) must be changed
quarterly (once every 3 months). For any lab device that contains <Company
Name> proprietary information, group account passwords must be changed
within three (3) days following a change in the group’s membership.
10. No lab shall provide production services. Production services are defined as
ongoing and shared business critical services that generate revenue streams
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 6 of 16
or provide customer capabilities. These should be managed by a <proper
support> organization.
11. InfoSec will address non-compliance waiver requests on a case-by-case
basis and approve waivers if justified.
3.2 General Configuration Requirements
1. All traffic between the corporate production and the lab network must go
through a Network Support Organization maintained firewall. Lab network
devices (including wireless) must not cross-connect the lab and production
networks.
2. Original firewall configurations and any changes thereto must be reviewed
and approved by InfoSec. InfoSec may require security improvements as
needed.
3. Labs are prohibited from engaging in port scanning, network auto-discovery,
traffic spamming/flooding, and other similar activities that negatively impact
the corporate network and/or non-<Company Name> networks. These
activities must be restricted within the lab.
4. Traffic between production networks and lab networks, as well as traffic
between separate lab networks, is permitted based on business needs and
as long as the traffic does not negatively affect other networks. Labs must
not advertise network services that may compromise production network
services or put lab confidential information at risk.
5. InfoSec reserves the right to audit all lab-related data and administration
processes at any time, including but not limited to, inbound and outbound
packets, firewalls and network peripherals.
6. Lab-owned gateway devices are required to comply with all <Company
Name> product security advisories and must authenticate against the
Corporate Authentication servers.
7. The enable password for all lab owned gateway devices must be different
from all other equipment passwords in the lab. The password must be in
accordance with <Company Name>'s Password Policy. The password will
only be provided to those who are authorized to administer the lab network.
8. In labs where non-<Company Name> personnel have physical access (e.g.,
training labs), direct connectivity to the corporate production network is not
allowed. Additionally, no <Company Name> confidential information can
reside on any computer equipment in these labs. Connectivity for authorized
personnel from these labs can be allowed to the corporate production
network only if authenticated against the Corporate Authentication servers,
temporary access lists (lock and key), SSH, client VPNs, or similar
technology approved by InfoSec.
9. Infrastructure devices (e.g. IP Phones) needing corporate network
connectivity must adhere to the Open Areas Policy.
10. All lab external connection requests must be reviewed and approved by
InfoSec. Analog or ISDN lines must be configured to accept only trusted call
numbers. Strong passwords must be used for authentication.
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 7 of 16
11. All lab networks with external connections must not be connected to
<Company Name> corporate production network or any other internal
network directly or via a wireless connection, or via any other form of
computing equipment. A waiver from InfoSec is required where air gapping is
not possible (e.g., Partner Connections to third party networks).
4.0 Enforcement
Any employee found to have violated this policy may be subject to disciplinary
action, up to and including termination of employment.
5.0 Definitions
1. Internal - A lab that is within <Company Name>'s corporate firewall and
connected to <Company Name>'s corporate production network
2. Network Support Organization - Any InfoSec approved <Company Name>
support organization that manages the networking of non-lab networks.
3. Lab Manager - The individual responsible for all lab activities and personnel
4. Lab - A Lab is any non-production environment, intended specifically for
developing, demonstrating, training and/or testing of a product.
5. External Connections (also known as DMZ) - External connections include
(but not limited to) third-party data network-to-network, analog and ISDN data
lines, or any other Telco data lines.
6. Lab Owned Gateway Device - A lab owned gateway device is the lab
device that connects the lab network to the rest of <Company Name>
network. All traffic between the lab and the corporate production network
must pass through the lab owned gateway device unless approved by
InfoSec.
7. Telco - A Telco is the equivalent to a service provider. Telcos offer network
connectivity, e.g., T1, T3, OC3, OC12 or DSL. Telcos are sometimes referred
to as "baby bells", although Sprint and AT&T are also considered Telcos.
Telco interfaces include BRI, or Basic Rate Interface - a structure commonly
used for ISDN service, and PRI, Primary Rate Interface - a structure for
voice/dial-up service.
8. Traffic - Mass volume of unauthorized and/or unsolicited network
Spamming/Flooding traffic
9. Firewall - A device that controls access between networks; it can be PIX,
that is, a router with access control lists or similar security devices approved
by InfoSec.
10. Extranet - Connections between third parties that require access to
connections non-public <Company Name> resources, as defined in InfoSec's
extranet policy (link).
11. DMZ (De-Militarized Zone) - This describes network that exists outside of
primary corporate firewalls, but are still under <Company Name>
administrative control.
6.0 Revision History
Technical enforcement
We have our policies, our scripts, or patch tool, but how to we enforce policy compliance? Within
the world of technology vendors, we are starting to see corporations adding tools to normal
network services. Both CISCO13 and Microsoft14 currently provide a Network Quarantine feature
to the systems that they offer. The offerings are both Server based and IP based approaches.
This technology is primarily for VPN connections at this time and is estimated to include normal
internal network traffic by 2005. The concept is that a computer system must pass a predefined
standard before they connect to the environment. Forrester recommends that you evaluate and
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 8 of 16
explore mixing technologies, however recent news has indicated that the two major players,
Microsoft and Cisco, may not be working with one another and you may have to choose.15
16
Graphical view of Microsoft Network Quarantine Services
Pete Boden, Director of US IT Corporate Security at Microsoft, used a TechNet Radio17 session
to discuss ways of ensuring that systems match the policies. At Microsoft, they perform tests that
Make sure the right version of Connection Manager is being used; do that to
make sure that all of the subsequent checks are in place and are current. Then
check for Windows XP Professional; that’s our remote-client-connection standard
OS, so everybody who connects remotely to our environment should be running
Windows XP Professional. Then we check that the Internet connection firewall is
enabled, that Internet connection sharing is disabled, and that the antivirus
product is loaded and all of the signature definitions are updated. After making
that connection, we launch a full compliance scan of the machine from our
network, which essentially runs through the rest of the patch and configuration
compliance checks. We make sure that the machine is fully patched; we make
sure that it’s configured correctly, and if it’s not, we deal directly with the asset
owner, the owner of that PC, to make sure that those checks or that those
standards are then met.
The compliance process starts with discovery and assessment. That means, first
of all we have to become aware that vulnerability exists. Typically, we use
mechanisms like the Microsoft Security Response Center, at least for Microsoft
vulnerabilities. Once we become aware of a vulnerability, we rate its severity to
determine how critical that vulnerability is and what we need to do to remediate.
If it is a critical vulnerability, we set a deadline for compliance, which is simply a
date by which every system on our network needs to comply with our patch
standard. The next step evaluates compliance across the network. As we set
those standards, and as we set the compliance deadline, we’re performing
baseline scans across our environment to see how compliant we already are. As
vulnerability information is released to our client base, many clients go out and
update themselves. They are allowed to update their own system and make sure
it’s compliant. At the same time we test to make sure that the patch is of high
enough quality to deploy, so we provide it to a very select group of application
owners to make sure that they can install that patch, run their business
appropriately, and uninstall it if they need to. Then we get to the deployment
phase, and this is the phase where we’re doing an enormous amount of
communication with our business systems owners and clients on the network to
make sure that they understand when they need to be patched by and what the
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 9 of 16
consequences are for noncompliance. Once we reach that patch deadline, we
drop into what we call the enforcement phase. Once we get to that phase, when
we identify a vulnerable system we take one of two measures. We either forcibly
patch or configure that machine or we remove it from the network.
Let’s stress this point again. These computers are your company assets; they do not belong to
the employees. Thus, setting policies and compliance is critical to maintain control.
Set the policy to make it easy to update and comply with
On the patch management listserve, people often ask for a template of all needed patches for a
workstation or server. What is disconcerting in this question is that there is a desire to rely only on
human interpretation of what is currently installed on a system. It’s much better to have a
combination of human expectation based on a review of the published bulletins, and a
reinforcement of this awareness by using a patch tool to confirm that “yes, you need to patch for
this.” While this book is sponsored by a patch management software solution vendor, we cannot
stress this point enough so we will make it loud and clear: If you rely solely on human
judgment for patching your systems, your risks are significantly increased and your rate
of failure is extremely high.
Whether you use free patch tools or develop your own scanning methodology, relying solely on
judgment and not following up with some sort of tool or technique to confirm your patch status is
extremely foolhardy. Patch and then CONFIRM the application and effectiveness of the patch. In
the windows environment confirm by checking registry keys or current, in-use DLLs to ensure
patch status. Utilities such as tlist, ListDLLs18 or Process Explorer19 assist in this process.
Stuff happens
Even with all of our testing, things happen. Therefore, ensure that you have good vendor
relationships. There is nothing so frustrating than the “ping pong” effect: Vendor A blames an
issue on Vendor B; Vendor B blames Vendor A. The more a specific vendor depends on your
business, the more power you have. While migrating to different vendors is not easy, the fact that
we are in an increasingly competitive marketplace helps greatly. On listserves and newsgroups,
we repeatedly see people post issues that they, clearly, could move to the vendor for resolution.
Repeatedly they are reluctant to do so. If cost is an issue, remember that calls to resolve issues
caused by a Microsoft security patch are free. When contacting any vendor, gather the following
information:
•
•
•
Details about the computer exhibiting the problem
o Version installed
o Service packs applied
o Affected application
ƒ Version
ƒ service pack
o Other applications on the box
o Antivirus software, version
o Processor
o Memory
o Hotfixes Installed
Details about other computers (a client machine, for example), if applicable
o Operating system and service pack
o Applications
o Antivirus software, version
Details about the network, if applicable
o Internet connection type
o Number of NICs in the server
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 10 of 16
•
o Firewall/proxy
o Network topology particulars
Details about the issue
o Full, complete error message wording or screenshot
o Timing of first occurrence
o Changes made around that time
o Steps taken to reproduce the error, if any
o Additional information you think might be helpful
There is a cartoon showing Bart Simpson writing on the blackboard, “I will search Google before
asking dumb questions.”20 You would be surprised how many people do not know the power of
the search engine. You can begin your error investigation process by using one of the greatest
diagnostic tools around: A search engine such as Google. Launch the Google search engine21
and put in your "exact" error message. You may be surprised that someone else had your same
problem and someone else has helped him or her find the answer. If that does not work, try
searching on Google Groups22, which is the search engine of the Usenet. To narrow your search
further, click on “Advanced” and limit your search to microsoft.com or redhat.com or the
appropriate vendor’s domain.
For narrowing down Usenet postings, click on advanced and include a wildcard, as in Microsoft.*,
which will search on the public newsgroups on the Microsoft domain. Lastly, after you review the
Event viewer, the web site of www.eventid.net is invaluable for quick resolution on security or
even non-security issues. Subscribe to the service. Those red stop signs in your event viewer and
the resulting code can point you in the direction of resolution.
Be willing to gather information
Without data, your vendor is dumb and blind to the issue you are seeing. If your system crashes
and asks to send a “Dr. Watson” dump to Microsoft, for example, send it. The information sent to
Microsoft or any vendor assists them greatly in pro-actively preventing issues. You may find that
doing a Dr. Watson error gets you a link to additional information for resolution. Next, have on
hand the tools that Microsoft Product Support or other vendors use in diagnosing issues. Tools
such at the PSS support tools23 allow engineers to more easily see what is going on. The support
engineer may also want you to use debugging programs24 to prepare a dump file for analysis.
Use additional tools from the Server resource kits25 as needed. The more information you provide
the faster and better will be the resolution. Finally, when you call in for support to any vendor,
make sure that you get to the right division or department.
Online communities offer another way to compare patch experiences. Phrase your subject line
carefully when sending a question and describe the steps that you have taken. Include the
relevant information to make the resolution process faster. Find additional guidance in asking
effective questions the right way at http://www.catb.org/~esr/faqs/smart-questions.html
Remember: There are no dumb questions. Thinking it is a dumb question is just dumb
because you will not find an answer until you ask the question. Always ask, as someone else
probably has the same question that you have.
Change Management follow-up
You’ve patched and you’ve enforced the policy to ensure that everyone patches. So now you are
done, right? Wrong. First, you need a change review process to summarize and gather those
items that testing missed. Can you cover an emerging issue in future testing procedures? Can
you better follow up with vendors and personnel? Did the patch affect one application in
particular? Use this review process to learn and make the next patch process easier.
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 11 of 16
Microsoft’s representation of the Change Management process
Determine which tools did the best job of recreating your network. If there were issues with
patches, what backup or imaging procedures worked the best? Did you ensure that during this
process, all appropriate change control mechanisms were in place? Use this process to learn and
refine your procedures as needed on an ongoing basis.
So I can’t patch, now what do I do?
There will be times when patching is either not an option or, in the case of zero day exploits
wherein the exploit is released to the web before a patch has been built. You might find that you
need to take alternative actions. Nevertheless, be sure to analyze carefully the true risk. Many
businesses that didn’t patch for 03-026 erroneously believed that they were safe because they
had a firewall; however, all MSBlaster (03-026) needed to shut down a business was a remote
connection in from an infected remote client. The recent addition of the XP SP2 firewall26
controlled from the INSIDE of a company helps limit internal exposure.
If contractual arrangements restrict you from patching, consider “air gaps” or other devices to limit
and restrict TCP/IP traffic. Vlans and virtual machines may not be enough to limit such traffic as
switches can be vulnerable devices as well. You can also use IPSec,27 which is “an
implementation of the Internet Engineering Task Force’s Internet Protocol security (IPSec).
IPSec, which is also included in Windows 2000 and Windows XP, provides network managers
with a key line of defense in protecting their networks. IPSec exists below the Transport layer, so
applications transparently inherit its security services. IPSec provides the protections of data
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 12 of 16
integrity, data origin authentication, data confidentiality, and replay protection without having to
upgrade applications or train users.”
IPSec provides defense-in-depth against vulnerabilities in upper-layer protocols and applications by
protecting upper layer protocols, services, and applications. With IPSec enabled, initial packets
access an application or service running on a server only if they meet the allowed policy. For
example, attempts to disable applications or services on servers must first penetrate the IPSec
protection. In addition, because packets will not be passed to the application or service until trust has
been established through IPSec authentication and the configured protection on packets for the
application or service is protected.
Use IPSec policy to restrict access to servers. You can configure a server to accept specific types
of traffic only. For example, you can configure an e-mail server to accept only secured e-mail
traffic from client computers. The e-mail server discards all other traffic from client computers.
What is the best way to find alternative to patching? Turn first to the security bulletins themselves.
Using our Security bulletin 03-02628 as an example, refer to the technical information section.
Scroll to the “Mitigating factors” area located in each bulletin. In this case, mitigation involved
blocking the affected ports. This is easy to do on a stand-alone machine, but hard to do in a
network.
The story of two patches
Two patches serve as examples of “major events” for IT professionals. Both were highly critical,
many administrators pushed both patches out to their network EXTREMELY fast. Moreover, while
one turned into a network event, the other seemed not quite the Internet threat that it initially
appeared to be.
Let’s look first at the MSBlaster security bulletin 03-02629 to review why it became an event:
Security bulletin 03-026 - Buffer Overrun In RPC Interface Could Allow Code Execution (823980)
MSBLAST
Issue
Impact of vulnerability
Result
Run code of attackers choice
Severity rating
Critical
Threat vectors
File and printing ports
Comment
“Run code” is translation for “oh this
is going to be bad”
Also critical on ALL platforms
[sometimes Windows 2003 has
mitigation issues when there are
security issues for Internet Explorer
due to the IE enhanced lockdown”
Network administrators thought they
had more control over the
parameters of their network. We
don’t. Plan accordingly.
Regarding Blaster, we knew that our firewall protected us on the outside. What we did not take
into account was how interconnected we are. The vulnerability for 03-026 spawned a worm called
MSBlaster. This worm got inside of our networks because of VPN and other remote connections.
It took down firms and put their servers and workstations offline. Even today, we remain at risk on
the inside; we cannot depend on our network perimeters alone. In-depth defense through
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 13 of 16
compliance with patch policies would have prevented MSBlaster. Now let us look at our second
sample patch:30
Security bulletin 04-011 - Buffer Overrun In RPC Interface Could Allow Code Execution (823980)
SASSER
Issue
Impact of vulnerability
Severity rating
Result
Remote code execution
Critical
Threat vectors
Primarily File and printing ports
Comment
Hmmm… this also sounds bad
But here’s the difference – NOT
critical on Windows 2003 servers
Primarily affected computers without
up to date firewalls and antivirus
Of all the issues addressed in this bulletin, the malware targeted only the LSASS service. The
mitigation of this one was clear:
An anonymous user can remotely attack Windows 2000 and Windows XP only.
While Windows Server 2003 and Windows XP 64-Bit Edition Version 2003
contain the vulnerability, only a local administrator could exploit it.
Given reviews of security issues, both patches deserved very quick installation, especially when
administrators compared notes and looked at the threat vectors. Many thought that the Sasser
worm would have exploited the SSL vulnerability instead of the LSASS service. Nevertheless,
each case highlights the same thing: The time between patch, exploit and event is shrinking.
The coming threats
Patching is just one source of protection. Forthcoming network protection technologies are just
the latest in the arsenal of tools that we must use to protect ourselves. Remember: it’s not just
about having a tool for patching; it’s about controlling the patching and enforcing patching best
practices. If you do not control your systems, they will control you.
We will complete our technical discussion of patches by looking at what lies ahead. A Microsoft
article31 seeks to:
• Establish a single set of patch related terms and mandate their use in all documentation
related to patches as described in knowledge base article 824684
• Establish the role, format, and content of knowledge base articles published in
conjunction with patches. The KB article will focus on installing the patch, while the
bulletin will focus on the vulnerability. as described in knowledge base article 824689
• Establish a common naming convention for all patch patches, providing information such
as the product the patch applies to, the language the patch was developed for, as
described in knowledge base article 824685
• Establish the standard that all patches use a properties page as discussed in knowledge
base article 824686
• Establish a single consistent way of recording a patch in the add/remove programs list.
Microsoft states that they are examining whether it may be appropriate to develop a
repository of information dedicated solely to patches
• Establish a standard such that all Microsoft patches will use either of two installer
technologies:
o Windows Installer [MSI] will be used by patches for applications
o Update.exe will be used by Windows operating systems and some applications
• Establish a standard of patch size reduction by removing debugging symbols from
patches. {If you receive a hotfix from Microsoft, you will notice for some hotfixes “symbol”
files inside the zipped hotfixes.
• Establish a uniform set of options that installer technologies will support as discussed in
knowledge base 824687
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 14 of 16
•
•
•
Establish a standard for uniform set of return codes and log entries that all patch
installers will use
Establish a standard allowing for uninstalling of all patches
Establish a standard of placing patches in a standard registry location
So what does the future hold for patch technology? Standardization is likely, and it can come
none too soon. As we begin to mix operating systems, applications, and platforms, standards
must emerge and evolve. Dig deep in your applications and work with your vendors to understand
how your systems work, to examine your systems and to understand how to patch them and how
to test the patches. Does this testing replicate your environment? The larger and more complex
your environment, the more ability you have to ask vendors to support and test patches before
you roll them out. Include such specifications in your vendor agreements. Begin to push, slowly at
first, so that your applications support the latest service packs within a reasonable period.
“Demand better security from vendors and hold them responsible. Use what you have, and make
sure you know how to use it properly and effectively”32
Ask your vendors to step up to the plate and support and test patches as well.
The bottom line for your boss
If you are reading this, chances are you are not the CEO or CIO or CFO. You probably do not
have budgetary control and the final say. Refer to the one-page appendix at the end of this
chapter. Copy it, hand it to the people who make the decisions in your firm. Take the time to
calculate the ALE formula. Discuss the impact of how the biggest security impact comes from
laptops that may not get the attention they need to maintain patch level. Take the time to review
the business side of patching to compile arguments you need to get the tools you must have.
Effective patch management is all about good process and good control. If you do not control
your server, your desktops, or your infrastructure; if you do not control who has access to your
network; and if you do not control what comes into and goes out of your network, you don’t
control anything. It is not YOUR network anymore.
The World has changed and computing environments continue to evolve. Companies that may
have considered security a minor compliance issue must now meet security standards. California
already has a law on the books commonly referred to as “SB138633.” This law requires a firm to
notify California residents when an unauthorized party views data sensitive to identity theft. While
this is an admirable law from the standpoint of the person susceptible to identity theft, it is a
potential liability and public relations nightmare. Management often requires ROI calculations on
new processes and purchases, but they often ignore intangible costs. The traditional ALE
equation may not give you the entire cost analysis you need but it may be all you have right now
to make your point.
CIO Magazine34 has predicted that the insurance industry will push for Security ROI and better
industry statistics, but that has yet to happen. We are still moving toward what Kevin Soo Hoo
described as “a place where computer security risks would be ‘actively and effectively
managed.’”35
Do you need to be empowered to take control of your network? Refer to the US CERT research
center for evidence on the survivability of the Internet.36 Consider, also, that Incidents.org, which
track events on the Internet, calculate that a Windows XP workstation can run on the world wide
web without protection for only twenty minutes.37 Ask yourself, if your Help Desk staff cannot get
to that system within that period and your employees control their systems and not your IT staff,
who really controls your network? It is certainly not your IT staff! Where is your weakest link?
Look toward the person who has the ability to download what he or she wants to. Patch
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 15 of 16
management is part of the process of having your IT staff stake a claim for once again controlling
YOUR network.
1
http://www.microsoft.com/technet/security/bulletin/MS04-011.mspx
http://www.microsoft.com/technet/security/bulletin/MS04-011.mspx
3
http://www.helpdesk.com/software-helpdesk.htm
4
http://support.microsoft.com/default.aspx?pr=securityitpro
5
http://bugzilla.redhat.com/bugzilla/query.cgi
6
http://www.ftponline.com/wss/2004_08/magazine/departments/guestop/
7
http://www.sans.org/resources/policies/
8
http://www.sans.org/resources/policies/Server_Security_Policy.doc
9
http://www.sans.org/resources/policies/Aquisition_Assessment_Policy.doc
10
http://www.sans.org/resources/policies/Audit_Policy.doc
11
http://www.sans.org/resources/policies/Virtual_Private_Network.doc
12
http://www.sans.org/resources/policies/
13
http://www.cisco.com/application/pdf/en/us/guest/netsol/ns466/c664/cdccont_0900aecd800ce939.pdf
14
http://www.microsoft.com/downloads/details.aspx?FamilyID=fe902704-52dd-4bbe-8a75f8fbb76cd28a&DisplayLang=en
15
http://searchnetworking.techtarget.com/originalContent/0,289142,sid7_gci992761,00.html
16
http://download.microsoft.com/download/0/7/e/07ed1953-0ab5-41ea-b5da41cf8bb9cdae/Quarantine.doc
17
http://www.microsoft.com/technet/community/tnradio/archive/tntrans_01.mspx
18
http://www.sysinternals.com/ntw2k/freeware/listdlls.shtml
19
http://www.sysinternals.com/ntw2k/freeware/procexp.shtml
20
http://dotnet.org.za/kevint/archive/2004/04/01/931.aspx
21
http://www.google.com/
22
http://www.google.com/grphp?hl=en&tab=wg&q=
23
http://www.microsoft.com/downloads/details.aspx?FamilyID=cebf3c7c-7ca5-408f-88b7f9c79b7306c0&DisplayLang=en
24
http://www.microsoft.com/whdc/devtools/debugging/default.mspx
25
http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69-57ff-4ae7-96eeb18c4790cffd&displaylang=en
26
http://www.microsoft.com/technet/prodtechnol/winxppro/maintain/xpsp2man.mspx
27
http://download.microsoft.com/download/5/d/b/5db098e2-43d3-48bc-9e5be72ab9c65c61/IPSec_Server2003.doc
28
http://www.microsoft.com/technet/security/bulletin/MS03-026.mspx
29
http://www.microsoft.com/technet/security/bulletin/MS03-026.mspx
30
http://www.microsoft.com/technet/security/bulletin/MS04-011.mspx
31
http://www.microsoft.com/technet/security/topics/patch/stdpatex.mspx
32
http://www.sbslinks.com/really.htm
33
http://info.sen.ca.gov/pub/01-02/bill/sen/sb_1351-1400/sb_1386_bill_20020926_chaptered.html
34
http://www.cio.com/archive/021502/security.html
35
http://www.sims.berkeley.edu/resources/affiliates/workshops/econsecurity/econws/06.doc
36
http://www.cert.org/nav/index_purple.html
37
http://isc.sans.org//survivalhistory.php
2
The Complete Patch Management Book
A Technical Exploration and Practical Guide
Chapter 4
Page 16 of 16