The Role of Operating Systems in Security
For every computer system and software design, it is imperative that it should address all security
concerns and implement required safeguards to enforce security policies. At the same time, it is
important to keep a balance since rigorous security measures can not only increase costs but also
limit the user-friendliness, usefulness and smooth performance of the system. Hence, system
designers have to ensure effective performance without compromising on security. A computer’s
operating system must concentrate on delivering a functionally complete and flexible set of
security mechanism for security policies to be effectively enforced.
An operating system’s protection and security requires all computer resources such as software,
CPU, memory and others to be protected. This can be enforced by ensuring the confidentiality,
integrity and availability in the operating system. It must be able to protect against all threats
including malware and unauthorized access.
Threats to Operating Systems
Let’s have a look at the common threats faced by any operating system.
Anything that has a malicious nature and can be harmful for the system is a threat.
Malware
This category includes viruses, worms, trojan horses and all kinds of malicious software. These
are generally small code snippets that can corrupt files, destroy data, replicate to spread further,
and even crash a system. Many times, the malware goes unnoticed by the victim user, while the
cyber criminals silently extract sensitive information.
Denial of Service Attacks
DDoS attacks run from a single IP address, while DDoS (distributed-denial-of-service) attacks are
carried out via numerous devices forming a botnet to increase the chances of the attack’s success.
With the growing number, complexity, and severity of DDoS attacks, it’s a good practice to
perform DDoS testing to check your operating system’s resilience to them.
Network Intrusion
Network intruders can be classified as masqueraders, misfeasors or a clandestine users. A
masquerader is an unauthorized individual who penetrates into a system and exploits an authorized
individual’s account. Misfeasor is a legitimate user who accesses and misuses programs, data or
resources. Clandestine user takes over supervisory control and tries to evade access controls and
audit collection.
Buffer Overflow
Also called buffer overrun, buffer overflow is defined in the NIST Glossary of Key information
security terms as “A condition at an interface under which more input can be placed into a buffer
or data-holding area than the capacity allocated, overwriting other information. Attackers exploit
such a condition to crash a system or to insert specially crafted code that allows them to gain
control of the system”
1
Buffer overflow is one of the most common and dangerous security threats. To exploit a buffer
overflow, attackers identify a buffer overflow vulnerability in a program and understand how the
buffer will store in process memory to finally alter the execution flow of the program.
Ensuring Operating Systems Security
Operating systems security can be ensured with the following mechanisms.
Authentication
Authentication identifies every user in a system and ensures that their identity is legitimate. The
operating system makes sure that each user is authenticated before they are allowed to access a
system. Different ways to ensure their authenticity are:
Username and Password
Every user has their distinct username and password that needs to be entered correctly before they
are able to access a system
User Attribution Identification
These methods usually involve biometrics verification such as finger prints, eye retina scan, etc.
This authentication is based on uniqueness of users and is compared with the database samples
that already exist in the system. Users can access only in case of a match.
One-Time Password
A one-time password is generated exclusively for each time a user wants to log in and enter a
system. The same password cannot be used again. Methods include:
 Random Numbers
The system may ask you for numbers corresponding to a set of pre-arranged alphabets. The
combination is different every time you require a login
 Secret Key
This includes a hardware device that generates a secret key for the user id, and changes every time.
Tokens
A user is authenticated with something that they physically possess, such as a smart card or
electronic keycard.
Access Control
Access control specifies who can have access to a system resource and what type of access each
entity has. A security administrator maintains an authorization database to specify what type of
access is allowed to each user. This database is consulted by the access control function for
determining whether access should be granted.
Intrusion Detection Systems
Intrusion Detection Systems monitor network traffic or events occurring within a host to identify
any suspicious activity. IDS helps identify network, transport and application protocols.
2
Firewalls
Firewalls are important to monitor all incoming and outgoing traffic. It enforces local security,
thus defining the traffic that is authorized to pass through it. Firewalls are effective means to
protect local systems or network of systems from all network-based security threats.
Buffer Overflow Defense
Countermeasures to avoid buffer overflow include compile-time defense, that aims to harden a
program for resisting an attack to enhance software security; or runtime defense, that detects and
aborts attacks in an executing program.
UNIX VULNERABILITIES
A secure operating system must protect its trusted computing base from compromise in order to
implement the reference monitor guarantees as well. In this section,we list some of the
vulnerabilities
that have been found in UNIX systems over the years that have resulted in the compromise of the
UNIX trusted computing base. This list is by no means comprehensive. Rather, we aim to provide
some examples of the types of problems encountered when the system design does not focus on
protecting the integrity of the trusted computing base.
Network-facing Daemons UNIX has several root (i.e., TCB) processes that maintain network
ports that are open to all remote parties (e.g., sshd, ftpd, sendmail, etc.), called network-facing
daemons. In order to maintain the integrity of the system’s trusted computing base, and hence
achieve the reference monitor guarantees, such process must protect themselves from such input.
However, several vulnerabilities have been reported for such processes, particularly due to buffer
overflows [232, 318], enabling remote attackers to compromise the system TCB. Some of these
daemons have been redesigned to remove many of such vulnerabilities (e.g., Postfix [317, 73] as
a
replacement for sendmail and privilege-separated SSH [251]), but a comprehensive justification
of
integrity protection for the resulting daemons is not provided.Thus, integrity protection of
networkfacing
dameons in UNIX is incomplete and ad hoc.
Further, some network-facing daemons, such as remote login daemons (e.g., telnet, rlogin,
etc.) ftpd, and NFS, puts an undo amount of trust in the network. The remote login daemons and
ftpd are notorious for sending passwords in the clear.Fortunately, such daemons have been
obsoleted
or replaced by more secure versions (e.g., vsftpd for ftpd). Also, NFS is notorious for accepting
any response to a remote file system request as being from a legitimate server [38]. Network-facing
daemons must additionally protect the integrity of their secrets and authenticate the sources of
remote data whose integrity is crucial to the process.
Rootkits Modern UNIX systems support extension via kernel modules that may be loaded
dynamically
into the kernel. However, a malicious or buggy module may enable an attacker to execute
3
code in the kernel, with full system privileges. A variety of malware packages, called rootkits,
have
been created for taking advantage of kernel module loading or other interfaces to the kernel
available
to root processes. Such rootkits enable the implementation of attacker function and provide
measures to evade from detection. Despite efforts to detect malware in the kernel [244, 245], such
rootkits are difficult to detect, in general,
Environment Variables UNIX systems support environment variables, system variables that are
available to processes to convey state across applications. One such variable is LIBPATH whose
value
determines the search order for dynamic libraries. A common vulnerability is that an attacker can
change LIBPATH to load an attacker-provided file as a dynamic library. Since environment
variables
are inherited when a child process is created, an untrusted process can invoke a TCB program
(e.g., a program file which setuid’s to root on invocation, see Section 4.2.2) under an untrusted
environment. If the TCB process depends on dynamic libraries and does not set the LIBPATH
itself,
it may be vulnerable to running malicious code. As manyTCB programs can be invoked via setuid,
this is a widespread issue.
Further,TCB programs may be vulnerable to any input value supplied by an untrusted process,
such as malicious input arguments. For example, a variety of program permit the caller to define
the configuration file of the process. A configuration file typically describes all the other places
that
the program should look for inputs to describe how it should function, sometimes including the
location of libraries that it should use and the location of hosts that provide network information.
If
the attack can control the choice of a program’s configuration file, she often has a variety of ways
to
compromise the running process. Any TCB program must ensure their integrity regardless of how
they are invoked.
Shared Resources If TCB processes share resources with untrusted processes, then they may be
vulnerable to attack. A common problem is the sharing of the /tmp directory. Since any process
can
create files in this directory, an untrusted process is able to create files in this directory and grant
other processes, in particular a TCB process, access to such files as well. If the untrusted process
can guess the name of TCB process’s /tmp file, it can create this file in advance, grant access to
the
TCB process, and then have access itself to a TCB file. TCB processes can prevent this problem
by checking for the existence of such files upon creation (e.g., using the O_CREAT flag).
However,
programmers have been prone to forget such safeguards. TCB process must take care when using
any objects shared by untrusted processes.
Time-of-Check-to-Time-of-Use (TOCTTOU) Finally, UNIX has been prone to a variety of
4
attacks where untrusted processes may change the state of the system between the time an
operation
is authorized and the time that the operation is performed. If such a change enables an untrusted
process to access a file that would not have been authorized for, then this presents a
vulnerability.The
attack was first identified by Dilger and Bishop [30] who gave it the moniker time-of-check-totimeofuse attacks or TOCTTOU attacks. In the classical example, a root process uses the system call
access to determine if the user for whom the process is running (e.g., the process was initiated by
a setuid) has access to a particular file /tmp/X. However, after the access system call authorizes
the file access and before the file open, the user may change the binding between the file name
and the actual file object (i.e., inode) accessed. This can be done by change the file /tmp/X to a
symbolic link to the target file /etc/shadow. As a result, UNIX added a flag, so the open request
could prevent traversal via symbolic links. However, the UNIX file system remains susceptible to
TOCTTOU attacks because the mapping between file names and actual file objects (inodes) can
be manipulated by the untrusted processes.
As a result of the discretionary protection system, the size of the system TCB, and these
types of vulnerabilities, converting a UNIX system to a secure operating system is a significant
challenge. Ensuring that TCB processes protect themselves, and thus protect a reference monitor
from tampering, is a complex undertaking as untrusted processes can control how TCB processes
are invoked and provide inputs in multiple ways: network, environment, and arguments. Further,
untrusted processes may use system interfaces to manipulate any shared resources and may even
change the binding between object name and the actual object.We will discuss the types of changes
necessary to convert an ordinary UNIX system to a system that aims to satisfy the secure operating
system definition in Chapters 7 and 9, so we will see that several fundamental changes are
necessary to
overcome these problems. Even then, the complexity of UNIX systems and their trusted computing
base makes satisfying the tamperproof and verifiability requirements of the reference monitor
concept
very difficult.
WINDOWS VULNERABILITIES
Not surprisingly given its common limitations,Windows suffers from the same kinds of
vulnerabilities
as the UNIX system (see Section 4.2.4). For example, there are books devoted to constructing
Windows rootkits [137].Herewe highlight a fewvulnerabilities that are specific toWindows
systems
or are more profound in Windows systems
TheWindows Registry TheWindows Registry is a global, hierarchical database to store data for
all programs [206].When a new application is loaded it may update the registry with
applicationspecific,
such as security-sensitive information such as the paths to libraries and executables to
be loaded for the application.While each registry entry can be associated with a security context
that limits access, such limitations are generally not effectively used. For example, the standard
configuration of AOL adds a registry entry that specifies the name of a Windows library file (i.e.,
5
DLL) to be loaded with AOL software [120]. However, the permissions were set such that any
user
could write the entry.
This use of the registry is not uncommon, as vendors have to ensure that their software will
execute when it is downloaded.Naturally, a user will be upset if she downloads some newlypurchased
software, and it does not execute correctly because it could not access its necessary libraries. Since
the
application vendors cannot know the ad hoc ways that aWindows system is administered, theymust
turn on permissions to ensure that whatever the user does the software runs. If the registry entry
is later used by an attacker to compromise the Windows system, that is not really the application
vendor’s problem—selling applications is.
Administrator Users We mentioned in the Windows security evaluation that traditionally users
ran under the identity Administrator or at least with administrative privileges enabled.The reason
for this is similar to the reason that broad access is granted to registry entries: the user also wants
to be
sure that they can use what function is necessary to enable the system to run. If the user downloads
some computer game, the user would need special privileges to install the game, and likely need
special privileges to run the device-intensive game program. The last thing the user wants is to
have
to figure out why the game will not run, so enabling all privileges works around this issue.
UNIX systems are generally used by more experienced computer users who understand the
difference between installing software (e.g., run sudo) and the normal operation of the
computer.As a result, the distinction between root users and sudo operations has been utilized more
effectively
in UNIX.
Enabled By Default Like users and software vendors,Windows deployments also came with full
permissions and functionality enabled. This resulted in the famous Code Red worms [88] which
attacked the SQL server component of the Microsoft IIS web server. Many people who ran IIS did
not have an SQL server running or even knew that the SQL server was enabled by default in their
IIS system. But in these halcyon times, IIS web servers ran with all software enabled, so attackers
could send malicious requests to SQL servers on any system, triggering a buffer overflow that was
the basis for this worm’s launch. Subsequent versions of IIS are now “locked down” 9, such that
software has to be manually enabled to be accessible.
Autoplay Vulnerability
Autoplay feature came in Windows XP. This feature checks removable media/ devices then
identifies and launches appropriate application based on its contents. This feature is useful for
authentic users but is a gateway for an attacker. The program developed was able to gain access
and execute arbitrary code by inserting USB using this feature.This vulnerability can be exploited
locally. The complexity of attack in this case is low. The system confidentiality and integrity is
lost completely.
Clipboard Vulnerability
The software developed was able to get access to clipboard data and modify it. This vulnerability
can allow attacker to get access to sensitive clipboard data. In windows clipboard is common for
6
all applications. This may lead to access and modification in the clipboard of all applications in
the operating system.
Registry Vulnerability
MS-Windows stores its configuration settings and options in a hierarchical database which is
known as windows Registry. Registry is used for low level operating system settings and for
settings of applications running on the platform. All vital components of operating system such as
kernel, UI, device drivers, SAM etc. make use of registry. The registry editor of windows is not a
secured program. It allows the editing of registries without the permission of the owner. As there
is no message specially displayed before editing of registry with software of executable files,
therefore the attackers are able to change the DWORD value of registry easily which poses a
serious threat
PNG Vulnerability
Software was able to cause denial of service (DoS attack). In this vulnerability Windows allows
an attacker to use Portable Network Graphic (PNG) image with properly crafted resolution in the
IHDR block which leads to 100% CPU consumption. Windows operating system is not equipped
to handle malicious PNG files. This vulnerability may result into excessive usage of resources and
causes system crash. Thus, denying service to users. This vulnerability does not result into
confidentiality or integrity loss and has partial availability impact
Result and Discussion
In the experiment, user was able to gain access through autorun vulnerability which is a serious
threat to the confidentiality and integrity of the same. Clip board vulnerability can also result into
severe damage to the data. Registry vulnerability can lead to unwanted operating system settings
by malicious user. PNG vulnerability causes denial of service and consumes resources. Microsoft
has still not released any patch for this vulnerability. Effect of these vulnerabilities was tested on
all popular versions of MS Windows like Windows XP, Windows Vista and Windows 7. Summary
of effect of these vulnerabilities is given in Table 1.
Operating system security (OS security) is the process of ensuring OS integrity, confidentiality and
availability.
OS security refers to specified steps or measures used to protect the OS from threats, viruses,
7
worms, malware or remote hacker intrusions. OS security encompasses all preventive-control
techniques, which safeguard any computer assets capable of being stolen, edited or deleted if OS
security is compromised.
OS security may be approached in many ways, including adherence to the following:

Performing regular OS patch updates

Installing updated antivirus engines and software

Scrutinizing all incoming and outgoing network traffic through a firewall

Creating secure accounts with required privileges only (i.e., user management)
Windows operating system security
Security
Measures
Features & Capabilities
Secure Boot and Secure Boot and Trusted Boot help prevent malware and corrupted components
Trusted Boot
from loading when a Windows device is starting. Secure Boot starts with initial
boot-up protection, and then Trusted Boot picks up the process. Together, Secure
Boot and Trusted Boot help to ensure your Windows system boots up safely and
securely.
Learn more Secure Boot and Trusted Boot.
Cryptography and Cryptography uses code to convert data so that only a specific recipient can read
certificate
it by using a key. Cryptography enforces privacy to prevent anyone except the
management
intended recipient from reading data, integrity to ensure data is free of
tampering, and authentication that verifies identity to ensure that communication
is
secure.
Learn
more
about Cryptography
8
and
certificate
management.
Security
Measures
Features & Capabilities
Windows Security The Windows built-in security application found in settings provides an at-aapp
glance view of the security status and health of your device. These insights help
you identify issues and take action to make sure you’re protected. You can
quickly see the status of your virus and threat protection, firewall and network
security,
device
security
controls,
and
more.
Learn more about the Windows Security app.
Encryption
and Wherever confidential data is stored, it must be protected against unauthorized
data protection
access, whether through physical device theft or from malicious applications.
Windows provides strong at-rest data-protection solutions that guard against
nefarious
attackers.
Learn more about Encryption.
BitLocker
BitLocker Drive Encryption is a data protection feature that integrates with the
operating system and addresses the threats of data theft or exposure from lost,
stolen, or inappropriately decommissioned computers. BitLocker provides the
most protection when used with a Trusted Platform Module (TPM) version 1.2
or
later.
Learn more about BitLocker.
Encrypted
Drive
Hard Encrypted Hard Drive uses the rapid encryption that is provided by BitLocker
Drive Encryption to enhance data security and management.
By offloading the cryptographic operations to hardware, Encrypted Hard Drives
increase BitLocker performance and reduce CPU usage and power consumption.
Because Encrypted Hard Drives encrypt data quickly, enterprise devices can
expand BitLocker deployment with minimal impact on productivity.
Learn
more
about Encrypted
Hard
Drives.
Security baselines A security baseline is a group of Microsoft-recommended configuration settings
that explains their security impact. These settings are based on feedback from
Microsoft security engineering teams, product groups, partners, and customers.
Security baselines are included in the Security Compliance Toolkit that you can
download
from
the
Microsoft
Download
Center.
Learn more about security baselines.
9
Features & Capabilities
Security
Measures
Virtual
Network
Private Virtual private networks (VPNs) are point-to-point connections across a private
or public network, such as the Internet. A VPN client uses special TCP/IP or
UDP-based protocols, called tunneling protocols, to make a virtual call to a
virtual
port
on
a
VPN
server.
Learn
more
about Virtual
Private
Networks.
Windows
Windows Defender Firewall is a stateful host firewall that helps secure the
Defender Firewall device by allowing you to create rules that determine which network traffic is
permitted to enter the device from the network and which network traffic the
device is allowed to send to the network. Windows Defender Firewall also
supports Internet Protocol security (IPsec), which you can use to require
authentication from any device that is attempting to communicate with your
device.
Learn more about Windows Defender Firewall with advanced security.
Antivirus
antimalware
protection
& Microsoft Defender Antivirus is included in all versions of Windows 10,
Windows Server 2016 and later, and Windows 11. If you have another antivirus
app installed and turned on, Microsoft Defender Antivirus will turn off
automatically. If you uninstall the other app, Microsoft Defender Antivirus will
turn
back
on.
From the moment you boot Windows, Microsoft Defender Antivirus continually
monitors for malware, viruses, and security threats. Updates are downloaded
automatically to help protect your device from threats. Microsoft Defender
Antivirus continually scans for malware and threats, and also detects and
blocks potentially unwanted applications (applications that can negatively
impact your device even though they are not considered malware).
Microsoft Defender Antivirus integrates with cloud-delivered protection, which
helps ensure near-instant detection and blocking of new and emerging threats.
Learn more about next-generation protection and Microsoft Defender Antivirus.
Attack
surface Your attack surfaces are the places and ways you are vulnerable to a cyber attack.
reduction rules
Attack surface reduction rules are built into Windows and Windows Server to
prevent and block certain behaviors that are often abused to compromise your
device or network. Such behaviors can include launching scripts or executables
10
Security
Measures
Features & Capabilities
that attempt to download or run other files, running suspicious scripts, or
performing other behaviors that apps don't typically initiate during normal work.
You can configure your attack surface reduction rules to protect against these
risky
behaviors.
Learn more about Attack surface reduction rules
Anti-tampering
protection
During cyber attacks (like ransomware attempts), bad actors attempt to disable
security features, such as antivirus protection on targeted devices. Bad actors
like to disable security features to get easier access to user’s data, to install
malware, or to otherwise exploit user’s data, identity, and devices without fear
of being blocked. Tamper protection helps prevent these kinds of activities.
With tamper protection, malware is prevented from taking actions such as:
Disabling
virus
and
threat
protection
Disabling
real-time
protection
Turning
off
behavior
monitoring
Disabling
antivirus
(such
as
IOfficeAntivirus
(IOAV))
Disabling
cloud-delivered
protection
Removing
security
intelligence
updates
Learn more about Tamper protection.
Network
protection
Network protection in Windows helps prevent users from accessing dangerous
IP addresses and domains that may host phishing scams, exploits, and other
malicious content on the Internet. Network protection is part of attack surface
reduction and helps provide an extra layer of protection for a user. Using
reputation-based services, network protection blocks access to potentially
harmful,
low-reputation
based
domains
and
IP
addresses.
In enterprise environments, network protection works best with Microsoft
Defender for Endpoint, which provides detailed reporting into protection events
as
part
of
larger
investigation
scenarios.
Learn more about Network protection.
Controlled folder With controlled folder access, you can protect your valuable information in
access
specific folders by managing apps’ access to specific folders. Only trusted apps
can access protected folders, which are specified when controlled folder access
is configured. Typically, commonly used folders, such as those used for
documents, pictures, downloads, are included in the list of controlled folders.
11
Security
Measures
Features & Capabilities
Controlled folder access helps protect valuable data from malicious apps and
threats,
such
as
ransomware.
Learn more about Controlled folder access.
Exploit protection Exploit protection, available in Windows 10, version 1709 and later,
automatically applies several exploit mitigation techniques to operating system
processes and apps. Exploit protection works best with Microsoft Defender for
Endpoint, which gives organizations detailed reporting into exploit protection
events and blocks as part of typical alert investigation scenarios.
You can enable exploit protection on an individual device, and then use Group
Policy to distribute the XML file to multiple devices simultaneously. When a
mitigation is encountered on the device, a notification will be displayed from the
Action Center. You can customize the notification with your company details
and contact information. You can also enable the rules individually to customize
which
techniques
the
feature
monitors.
Learn more about Exploit protection.
Microsoft
Defender
Endpoint
Windows E5 customers benefit from Microsoft Defender for Endpoint, an
for enterprise endpoint detection and response capability that helps enterprise
security teams detect, investigate, and respond to advanced threats. With rich
event data and attack insights, Defender for Endpoint enables your security team
to investigate incidents and take remediation actions effectively and efficiently.
Defender for Endpoint also is part of Microsoft 365 Defender, a unified pre- and
post-breach enterprise defense suite that natively coordinates detection,
prevention, investigation, and response across endpoints, identities, email, and
applications to provide integrated protection against sophisticated attacks.
Learn more about Microsoft Defender for Endpoint and Microsoft 365
Defender.
12