Guide to Operating
System Security
Chapter 3
Security Through
Authentication and
Encryption
Objectives




Explain encryption methods and how they are
used
Describe authentication methods and how they
are used
Explain and configure IP Security
Discuss attacks on encryption and
authentication methods
Guide to Operating System Security
2
Encryption





Uses a secret code to disguise data
Makes data unintelligible to everyone except
intended recipients
Protects data from attackers using a sniffer
Uses cryptography
Typically involves a key and an algorithm
Guide to Operating System Security
3
Encryption Methods (Continued)






Stream cipher and block cipher
Secret key
Public key
Hashing
Data encryption standard (DES)
RSA encryption
Guide to Operating System Security
4
Encryption Methods (Continued)




Pluggable authentication modules (PAMs)
Microsoft Point-to-Point Encryption (MPPE)
Encrypting File System (EFS)
Cryptographic File System (CFS)
Guide to Operating System Security
5
Stream Cipher and Block Cipher

Stream cipher


Every bit in a stream of data is encrypted
Block cipher


Encrypts groupings of data in blocks
Typically has specific block and key sizes
Guide to Operating System Security
6
Secret Key


Keeps encryption key secret from public
access, particularly over a network connection
Uses symmetrical encryption (same key to
encrypt and decrypt)
Guide to Operating System Security
7
Public Key


Uses public key and private key combination
(asymmetric encryption)
Public key can be communicated over an
unsecured connection
Guide to Operating System Security
8
Hashing

Uses one-way function to mix up message
contents





Scrambles message
Associates it with a unique digital signature
Enables it to be picked out of a table
Often used to create a digital signature
Hashing algorithms work on only one side of a
two-way communication
Guide to Operating System Security
9
Typically Used Hashing
Algorithms


Message Digest 2 (MD2)
Message Digest 4 (MD4)




MS-CHAP v1
MS-CHAP in Windows Server 2003
Message Digest 5 (MD5)
Secure Hash Algorithm 1 (SHA-1)
Guide to Operating System Security
10
MS-CHAP v1 or MS-CHAP
Encryption
Guide to Operating System Security
11
Data Encryption Standard



Developed by IBM; refined by the National
Bureau of Standards
Originally developed to use a 56-bit encryption
key
New version: 3DES (Triple DES)


Hashes data three times
Uses a key of up to 168 bits in length
Guide to Operating System Security
12
Using DES with IPSec in Windows
Server 2003
Guide to Operating System Security
13
Advanced Encryption Standard



Adopted by U.S. government to replace DES
and 3DES
Employs private-key block-cipher form of
encryption
Employs an algorithm called Rijndael
Guide to Operating System Security
14
RSA Encryption




Uses asymmetrical public and private keys
along with an algorithm that relies on factoring
large prime numbers
The algorithm uses a trapdoor function to
manipulate prime numbers
More secure than DES and 3DES
Used in Internet Explorer and Netscape
Navigator
Guide to Operating System Security
15
Pluggable Authentication
Modules


Can be installed in UNIX or Linux OS without
rewriting and recompiling existing code
Enable use of encryption techniques other than
DES for passwords and communications on a
network
Guide to Operating System Security
16
Microsoft Point-to-Point
Encryption


Used by Microsoft operating systems for
remote communications over PPP or PPTP
Uses RSA encryption



Basic encryption (40-bit key)
Strong encryption (56-bit key)
Strongest encryption (128-bit key)
Guide to Operating System Security
17
Encrypting File System


Set by an attribute of Windows OSs that use
NTFS
Protects folder/file contents on hard disk



Enables user to encrypt contents of folder/file so it
can only be accessed via private key code by user
who encrypted it
Employs DES for encryption
Uses a registered recovery agent
Guide to Operating System Security
18
How to Configure EFS


As an advanced folder attribute
By using cipher command in Command
Prompt window
Guide to Operating System Security
19
Configuring EFS as an
Advanced Folder Attribute
Guide to Operating System Security
20
Cipher Command-Line Parameters
Guide to Operating System Security
21
Cryptographic File System


File system add-on available as open source
software for UNIX and Linux systems
Enables encryption of disk file systems and
NFS files
Guide to Operating System Security
22
Summary of Encryption
Techniques (Continued)
Guide to Operating System Security
continued… 23
Summary of Encryption
Techniques (Continued)
Guide to Operating System Security
24
Authentication




Process of verifying that a user is authorized to
access particular resources
Typically associated with logon process
Validates both user account name and
password before giving access to resources
Often uses encryption techniques to protect
user names and passwords
Guide to Operating System Security
25
Authentication Methods (Continued)






Session authentication
Digital certificates
NT LAN Manager
Kerberos
Extensible Authentication Protocol (EAP)
Secure Sockets Layer (SSL)
Guide to Operating System Security
26
Authentication Methods (Continued)



Transport Layer Security (TLS)
Secure Shell (SSH)
Security token
Guide to Operating System Security
27
Session Authentication


Ensures packets can be read in correct order
Provides a way to encrypt the sequence order
to discourage attackers
Guide to Operating System Security
28
Digital Certificate




Set of unique identification information
typically put at the end of the file or associated
with a computer communication
Shows that the source of the file or
communication is legitimate
Typically encrypted by a private key and
decrypted by a public key
Issued by a certificate authority
Guide to Operating System Security
29
Digital Certificate Contents







Version
Certificate serial number
Signature algorithm identifier
Name of issuer
Validity period
Subject name
Subject public key information
Guide to Operating System Security
30
NT LAN Manager


Form of session authentication and
challenge/response authentication compatible
with all Microsoft Windows operating systems
Challenge/response authentication


Hashes an account’s password
Uses a secret key
Guide to Operating System Security
31
Kerberos

Employs private-key security and use of
tickets that are exchanged between the client
who requests logon and network services
access and the server, application, or directory
service that grants access
Guide to Operating System Security
32
Kerberos Configuration Options
Guide to Operating System Security
33
Extensible Authentication
Protocol



Multipurpose authentication method used on
networks and in remote communications
Can employ many encryption methods (DES,
3DES, public key encryption, smart cards, and
certificates)
Typically provides an authentication
communication between a computer and a
server used to authenticate computer’s access
Guide to Operating System Security
34
Secure Sockets Layer




Service-independent; broad uses for
e-commerce, HTTP, HTTPS, FTP, SMTP, and
NNTP
Developed by Netscape
Uses RSA public-key encryption
Most commonly used form of security for
communications and transactions over the Web
Guide to Operating System Security
35
Transport Layer Security


Modeled after SSL
Uses private-key symmetric data encryption
and TLS Handshake Protocol
Guide to Operating System Security
36
Secure Shell

Developed for UNIX/Linux systems to provide
authentication security for TCP/IP
applications, including FTP and Telnet
Guide to Operating System Security
37
Using Secure Shell
Guide to Operating System Security
38
Security Token


Physical device, often resembling a credit card
or keyfob, used for authentication
Communicates with an authentication server to
generate the password, using encryption for
exchange of password-generating information
Guide to Operating System Security
39
Advantages of Security Token


User does not have to memorize password
Value of password only lasts as long as the
communications session; new password is
created next time the security token is used
Guide to Operating System Security
40
Guide to Operating System Security
41
IP Security (IPSec)



Set of IP-based secure communications and
encryption standards developed by the IETF
Protect network communications through IP
Elements that enable security measures


Authentication header
Encapsulating Security Payload (ESP)
Guide to Operating System Security
42
IPSec Security Roles
Guide to Operating System Security
43
Authentication Header (AH)


Ensures integrity of a data transmission
Ensures authentication of a packet by enabling
verification of its source
Guide to Operating System Security
44
Specific Fields in AH






Next header
Payload length
Reserved
Security Parameter Index (SPI)
Sequence number
Authentication Data
Guide to Operating System Security
45
Encapsulating Security Payload
(ESP)



Encrypts packet-based data
Authenticates data
Generally ensures security and confidentiality
of network layer information and data within
packet
Guide to Operating System Security
46
Specific Fields in ESP







Security Parameter Index (SPI)
Sequence number
Payload data
Padding
Pad length
Next header
Authentication date
Guide to Operating System Security
47
Attacks on Encryption and
Authentication
Guide to Operating System Security
48
Guidelines for Resisting Attacks





Use strong passwords
Use strongest forms of authentication and
encryption permitted by OS
Use longest encryption keys possible
Inventory encryption and authentication
methods used by OS; close any holes
Have administrators use personal accounts
with administrative privileges (rather than use
administrative account directly)
Guide to Operating System Security
49
Summary





Encryption methods and how operating
systems use them
How systems authenticate one another
How to configure Kerberos authentication
logon security
How to use IP security to keep your TCP/IP
network secure
Typical methods attackers use to defeat
encryption and authentication
Guide to Operating System Security
50