Instructor: Prof. Michael P. Harris, CCNA CCAI
ITSY 2400 – Operating Systems Security
Chapter 6
Firewalls and Border Security
Operating Systems Security - Chapter 6
Firewalls and Border Security
Chapter Overview
In this chapter, we will learn the basics about the common language of networks, the TCP &
UDP (OSI Layer 4), and IP (OSI Layer 3) protocols. Learning about these protocols enables us
to understand their security vulnerabilities and how these can be mitigated. We will review
IP addressing, including how it can be used to thwart attacks. You will also be introduced to
border and firewall security, which can use characteristics of TCP, UDP, and IP to build more
secure networks. Finally, you’ll learn how to configure the firewall capabilities of OS.
Learning Objectives
After reading this chapter and completing the exercises, students will be able to:
 Understand how TCP, UDP, and IP work and understand their security vulnerabilities
 Explain the use of IP addressing on a network and how it is used for security
 Explain border and firewall security
 Configure the firewall capabilities in operating systems
Lecture Notes
An Overview of TCP, UDP, and IP
Since its introduction in the early 1970s, Transmission Control Protocol / Internet Protocol,
or TCP/IP, has been widely used on networks throughout the world. It is the networking
protocol of choice for modern Windows, UNIX/Linux, and NetWare systems. TCP/IP contains
nearly 100 open protocols that interconnect computer systems efficiently & reliably.
The core component protocols within the TCP/IP protocol suite are:
 Transmission Control Protocol (TCP)
 User Datagram Protocol (UDP)
 Internet Protocol (IP)
Understanding Transmission Control Protocol (TCP)
Transmission Control Protocol (TCP) is a transport protocol that maintains communication
sessions between software application processes initiated by users on a network. TCP
provides for reliable end-to-end delivery of data by monitoring the accurate receipt of
packets and by controlling data flow. TCP accomplishes this by sequencing and
acknowledging packets, both characteristics that enhance security as part of a connectionoriented services approach to communications.
Michael Palmer, GUIDE TO Operating Systems Security
Thompson/Course Technology ©2004
Page 1 of 7
ISBN: 0-619-16040-3
Instructor: Prof. Michael P. Harris, CCNA CCAI
ITSY 2400 – Operating Systems Security
Chapter 6
Firewalls and Border Security
The TCP frame contains a header and payload data (see Figure 6-1 on page 258 of the text)
and is called the TCP segment. The TCP header is a minimum of 20 bytes in length and
contains the following fields:
 Source port

Destination port
 Sequence number

Acknowledgement number
 Offset or header length

Flag/control
 Window

Checksum
 Urgent pointer

Options
 Padding
Attackers use this knowledge to scanning networks and launch attacks. TCP & UDP portscanning software may be used to simply collect information about a target, without the
target’s knowledge; gain access to a system, or used to crash a system.
An intruder may simply collect open ports/sockets information for later use, and not
connect. An attacker can use port-scanning software to overrun ports at the target with
repeated packets containing the SYN code bit, to establish a communication, and then send
repeated RST code bits to prevent immediate responses from the target.
Understanding User Datagram Protocol (UDP)
One limitation of TCP is that its connection-oriented design can create overhead on a busy
network. User Datagram Protocol (UDP) can be used as an alternative to TCP for
communications that do not require the same level of reliability (handshaking) as provided by
TCP. When it transmits data, the TCP/IP suite has the option to transmit data using UDP
instead of TCP. UDP employs connectionless services (no handshaking)with no reliability
checks such as sequencing and acknowledgements and containing virtually no overhead on top
of the IP-based datagrams sent. The UDP header has the following fields:
 Source port

Destination port
 Length

Checksum
Because it does not use sequencing and acknowledgements, UDP is simpler than TCP and
port-scanning attacks are less effective. A UDP port can appear open to a port scanner,
when it is really closed. This is because the target may –or may not send back an Internet
Control Message Protocol (ICMP) message indicating that the port cannot be reached.
Understanding How the Internet Protocol (IP) Works
An enterprise may be composed of a series of subnetworks. The IP-address/subnet mask
enables a packet to reach different subnetworks on a LAN and different networks on a WAN.
The Basic Functions of IP
The basic functions of IP are to provide for data transfer, packet addressing, packet routing,
fragmentation control, and simple detection of packet errors. Each network host has a
32-bit IP-address (in IPv4), which, when used with its 48-bit Media Access Control (MAC)
address, enables network (LAN) and internetwork (WAN) communications and accurate
delivery of packets. The MAC address, sometimes called the Burned-In Address (BIA), is
permanently burned into the Network Interface Card (NIC).
Michael Palmer, GUIDE TO Operating Systems Security
Thompson/Course Technology ©2004
Page 2 of 7
ISBN: 0-619-16040-3
Instructor: Prof. Michael P. Harris, CCNA CCAI
ITSY 2400 – Operating Systems Security
Chapter 6
Firewalls and Border Security
IP as a Connectionless Protocol
IP is a connectionless protocol because its primary mission is to provide network-to-network
addressing and routing information, and to change the size of packets when the size varies
from network to network. When the OSI Layer 4, TCP segment, is encapsulated with the
additional OSI Layer 3, IP header information, the entire unit is called a datagram or
packet, as shown in Figure 6-3 on page 264.
The IP packet header consists of the following fields, as shown in Figure 6-4 on page 265:
 Version

IP header length (IHL)
 Type of service (TOS)

Length
This list continues on pages 264 through 266 of the text.
How IP Addressing Works
IP addressing is used to identify a specific host and the network on which it resides. The IP
address format is called the dotted decimal notation address. It is 32 bits long and contains
four octets, which are decimal values representing 8-bit bytes. An IP address in binary looks
like this: 10000001.00000101.00001010.01100100. This converts to 129.5.10.100
There are five IP address classes, Class A through Class E, each used with a different type
network. Classes A through C are intended for normal unicast addressing, but each class
represents a different network size.
Using a Subnet Mask
IP addresses require a subnet mask. A subnet mask is used for two purposes: to determine
how portions of addresses on a network are divided into the network ID and the host ID, and
if needed -divide a network into subnetworks to control network traffic.
Creating Subnetworks
To divide the network into subnets, a classless subnet mask determined by the network
administrator, divides a network into subnetworks and valid host ID ranges. Using a subnet
mask to divide a network into a series of smaller networks enables routing devices to
effectively ignore traditional address class designations (classful), and therefore creates more
options for segmenting networks through multiple subnets and additional network addresses,
to overcome the classful network size limitation in IPv4. A newer way to specify the classless
addressing (subnetting) subnet mask is by using the Classless Inter-Domain Routing (CIDR)
notation, which puts a slash (/) after the dotted decimal notation followed by the number
of 1-bits specified in the subnet mask. Subnetting provides more IP-address options for
medium-sized networks, because there is a shortage of Class B and Class C addresses. In the
Hands-on Projects 6-1 through 6-4 on pages 298 through 301 you will learn how to determine
IP address/Subnet Mask information for Windows, Linux, NetWare, and Mac OS X.
Border and Firewall Security
A Border (Border Gateway) is typically established between a private network (enterprise)
one used by a company for exampleand a public network, in particular the Internet.
For security, organizations establish border gateways at each border crossing. The border
gateway is a firewall that is configured with security policies to control the traffic that is
permitted to cross a border in either direction.
Michael Palmer, GUIDE TO Operating Systems Security
Thompson/Course Technology ©2004
Page 3 of 7
ISBN: 0-619-16040-3
Instructor: Prof. Michael P. Harris, CCNA CCAI
ITSY 2400 – Operating Systems Security
Chapter 6
Firewalls and Border Security
The strongest border security design is to protect every border point, including:
 Connection points between LANs and public or private WANS
 Dial-up and cable modem access
 Virtual private network (VPN) access
 Short-range wireless access, including 802.11 Wireless and Bluetooth
 Long-range wireless access, including satellite and microwave
Firewalls provide border security by using some or all of the following approaches:
 Packet filtering
 Network Address Translation (NAT)
 application gateways or proxies
Packet Filtering
Packet filtering typically involves using characteristics of TCP (or UDP) and IP to establish
filters between two connected networks. Another type of packet filtering is to allow or block
packets from specific protocols. For example, a packet filter might block NetBEUI protocol
packets from an older Windows NT Server network, or it might block Internet Packet
Exchange (IPX) protocol packets used by an older NetWare network. The IPX protocol was
developed by Novell and was used extensively for versions of NetWare prior to version 5. A
disadvantage of IPX is that it is a “chatty” protocol, because computers that use it frequently
broadcast “I’m here” (hello) messages that can cause significant network traffic.
When you create a filter for TCP/IP, two important characteristics are the IP address
information in a packet and the TCP or UDP port (socket) information. Another way to use a
firewall is to control access across the firewall by TCP and/or UDP port number. Figure 6-7
on page 274 illustrates the use of a firewall to protect a specific subnet via the subnet
identification and through port blocking.
Packet filtering is accomplished using one of two techniques: stateless filtering and
stateful filtering. In stateless packet filtering the firewall examines every individual packet
and decides whether to pass or block the packet, depending on the packet or segment header
information. Stateful packet filtering, also called Stateful Packet Inspection (SPI), tracks
information about a communication session, such as which ports/sockets are in use, drawing
from the contents of multiple packets.
Network Address Translation (NAT)
When NAT is used, private IP-addresses on the network protected by NAT are seen by the
outside world as a single public IP-address, the WAN IP-address of the device configured for
NAT, or as a public IP-address selected from a pool of real but proxy IP-addresses. Using NAT
discourages attackers, because they cannot identify a specific host to attack behind the NAT
firewall device on the local internal network. Instead, the attacker sees only the global
external IP-address used by the device running the NAT firewall software (usually a router).
Another advantage of NAT is that it enables a network to use IP-addresses on the internal
local network that are not formally registered for Internet use (Private IP-Addresses). There
are generally four ways to perform NAT translation:
 Dynamic translation (or IP masquerade)

Static translation
 Network redundancy translation

Load balancing
Michael Palmer, GUIDE TO Operating Systems Security
Thompson/Course Technology ©2004
Page 4 of 7
ISBN: 0-619-16040-3
Instructor: Prof. Michael P. Harris, CCNA CCAI
ITSY 2400 – Operating Systems Security
Chapter 6
Firewalls and Border Security
Proxies
A proxy is a host computer that is located between a host on an internal network and a host
on an external network with which the internal host is communicating. A proxy can fulfill
one or a combination of tasks:
 Act as an application-level gateway
 Filter communications
 Create secure tunnels for communications
 Enhance application request performance through caching
Proxies that are configured as application-level gateways can have different levels of
ability. Some proxies function as circuit-level gateways, creating a virtual tunnel or
Virtual Private Network (VPN) between the proxy and an external host. Some proxies are
able to provide caching services as a way to reduce the load on servers within the internal
network. Cache is storage used by a computer system to house frequently-used data in
quickly accessed storage, such as memory.
Using Routers for Border Security
A router performs packet filtering and is often used as a policy firewall on a network, in
addition to the other functions it performs. In general, routers are used to:
 Efficiently direct packets from one network to another (routing)
 Join neighboring or distant networks (VPNs)
 Connect dissimilar networks (gateway)
 Prevent network bottlenecks by isolating portions of a network (QoS)
 Secure portions of a network from intruders (IDS)
 Permit or deny packets on the basis of the source IP, the destination IP, the protocol
(port/socket) used, and whether the packet is inbound or outbound (ACL)
A Routers routing protocol (RIP, OSPF, BGP …) regularly exchanges information about
network traffic, the network topology, and the status of network links. When a packet
arrives, the router examines the protocol destination address, for instance the IP-address in a
TCP/IP packet. The router then determines how to forward (route) the packet on the basis
of the metrics it uses in the routing table (best path). A metric (cost value) is used to
determine the best path/route through an inter-network (WAN).
Routers that are in a local systemfor example, within a single organization and on the same
WAN (enterprise)use two common routing protocols for communications: RIP and OSPF.
Routing Information Protocol (RIP) is used by routers to determine the fewest hops between
themselves and other routers, and this information is added to each router’s routing table.
Open Shortest Path First (OSPF) is more commonly used and offers several advantages over
RIP. One advantage is that the router sends only the portion of the routing table that
pertains to its most immediate change in router links. This is called the “link-state routing.”
Two other advantages of OSPF are:
 It packages routing information in a more compact packet format than RIP
 Update routing link information is shared among routers, rather than the entire
routing table.
Michael Palmer, GUIDE TO Operating Systems Security
Thompson/Course Technology ©2004
Page 5 of 7
ISBN: 0-619-16040-3
Instructor: Prof. Michael P. Harris, CCNA CCAI
ITSY 2400 – Operating Systems Security
Chapter 6
Firewalls and Border Security
Using the Firewall Capabilities in Operating Systems
Some operating systems allow you to configure firewall services for border security. A
demilitarized zone (DMZ) is a separate portion of a network (a subnet) that exists between
two or more networks that have different security measures in place, such as the “zone”
between the private network of a company and the Internet. The advantage of placing
servers in the DMZ is that the less-secure network communications required for access to the
servers does not have to cross the border into the private network.
Configuring a Firewall in Windows XP Professional
When a host running Windows XP Professional is directly connected to the Internet, through
a cable modem or DSL connection, the Internet Connection Firewall (ICF) should be enabled.
See Figure 6-11 on page 284. You can also configure ICF for Internet Connection Sharing
(ICS) (e.g. NAT) for hosts on a local area connection, particularly if these hosts are not
already protected by a firewall. Once ICF is enabled, you can choose to allow or deny
incoming services (protocol/port/socket), such as HTTP, HTTPS, FTP, SMTP, and others.
Quick Reference
Discuss what ICF is designed to do when it is enabled as shown on page 284.
Configuring a Firewall in Windows Server
Windows Server 2003/2008 uses the same implementations of ICF as in Windows XP
Professional. When you configure ICF for Windows Server, make sure that you enable only
those services that are needed on the server; for instance, enable HTTP if you access webpages on the Internet.
Not all versions of Windows Server 2003 come with ICF. It is packaged with Windows Server
2003, Standard Edition, and Windows Server 2003, Enterprise Edition. ICF is not available in:
 64-bit versions of Windows Server 2003
 Windows Server 2003, Datacenter Edition
 Windows Server 2003, Web Edition
Configuring NAT in Windows Server 2003/2008
Windows Server 2003/2008 can be configured to provide NAT firewall services, for
connections that go over the Internet. NAT is just one of several services that can be set up
in Windows Server through Microsoft Routing and Remote Access Services (RRAS) (see Figure
6-13 on page 286). When you configure NAT in Windows Server 2003/2008, you can configure
it to work with one or more NICs –to connect LAN(s) or WAN connections to the server, or
both.
Configuring NAT in Windows 2000 Server
In Windows 2000 Server, you can enable NAT by setting up the Windows 2000 server as an
Internet connection server in the Windows 2000 Server Routing and Remote Access tool.
When you configure Windows 2000 Server to use NAT, it functions similarly to the NAT
implementation for a small office described for Windows Server 2003. Hands-on Project 6-7
on pages 304 and 305 shows how to configure NAT in Windows 2000 Server.
Michael Palmer, GUIDE TO Operating Systems Security
Thompson/Course Technology ©2004
Page 6 of 7
ISBN: 0-619-16040-3
Instructor: Prof. Michael P. Harris, CCNA CCAI
Chapter 6
Firewalls and Border Security
ITSY 2400 – Operating Systems Security
Configuring a Firewall in Linux
The simplest way to configure a firewall in Red Hat Linux is by using the Security Level
Configuration tool. This tool offers three basic security levels:
 High

Medium

No Firewall
Additionally, when you customize the firewall,
combination of the following services:
 WWW (HTTP)

 SSH

 Mail (SMTP/POP3/IMAP)

you can allow or deny access to any
FTP
DHCP
Telnet
Configuring NAT and a Firewall Using iptables in Linux
Linux also offers the powerful iptables capability for configuring NAT and complex firewall
security from the command line in a terminal window. If you are configuring a server or you
want to fine-tune your firewall security on a workstation, configure the firewall using
iptables instead of using the more basic Security Level Configuration tool. Iptables enables
configuration of packet filter rules. A set of rules is called a chain, and it is applied to
packets containing specific information. Table 6-3 on page 289 shows a sample of the
parameters that you can use with the iptables command.
Configuring a Mac OS X Firewall
Mac OS X comes with a firewall that you can configure to control access into and out of the
operating system over a network or Internet connection. The Mac OS X firewall enables you
to allow or deny network communications through TCP and UDP ports by first turning specific
services on or off. The services that you can turn on or off are:
 Personal file sharing

Windows file sharing
 Personal Web sharing
 Remote login – SSH
 FTP access

Remote Apple events
 Printer sharing
You can turn the firewall on or off, so that it allows or denies incoming network
communications to the configured services.
Discussion Questions
1) Discuss the importance and functionality of a subnet mask
2) Discuss the procedures used to configure an iptables firewall in Linux
Additional Activities
1) Have students chart the differences and similarities between the commonly used
network protocols.
2) Have students chart the differences and similarities of configuring NAT for Linux and
the Windows operating systems
Michael Palmer, GUIDE TO Operating Systems Security
Thompson/Course Technology ©2004
Page 7 of 7
ISBN: 0-619-16040-3