Chapter 3: Program Security
The theme of the third chapter seems to be very simply that low-quality software is less
secure than well written software. The author of these notes has no quarrel with that claim;
there is just too much evidence for it. The topics for this chapter fall into four categories.
1.
2.
3.
4.
Programming errors with security implications – mostly buffer overflows.
Malicious code that often targets these errors, including viruses, worms, etc.
Software engineering practices that might mitigate security issues.
Controls against program threats.
Bugs, Errors, and Security Flaws
Security faults in a computer program are only one type of error, although one can postulate
that any program error can be leveraged into a security flaw. Regrettably, there is a culture
in the development of software that all programs will contain errors and that there is nothing
to be done about it. This attitude appears excessively defeatist, although it seems to reflect
not only the actual state of affairs but also any reasonably expected state of affairs.
One of the most insidious aspects of software quality control is the “software bug”. The
term seems to have originated in one of the early (1940’s era) relay-operated computers when
a moth jammed a relay in the open position, causing the program to malfunction. The moth
was removed and taped into a research notebook with the caption “program bug”. One
wonders if the term gained widespread acceptance as a psychological crutch: “The program
did not have any bugs in it when I developed it, they must have crept in when nobody was
looking”. Yea, sure.
The major issue in the development of quality software appears to be proper software testing.
The old “penetrate and patch” or “tiger team” approaches remain valid, but need to be
supplemented by other methods. We shall return to the topic of software testing when we
discuss software engineering a bit later.
Program errors, even those representing security vulnerabilities, are normally considered to
be non-malicious in that they were not deliberately generated. We shall now consider a few
of these non-malicious errors, some due either to laziness or pressures to develop on time.
Buffer Overflows
The software error that most commonly leads to security vulnerabilities seems to be that of
buffer overflows. This occurs when a buffer is allocated to hold data and the code that
places data into the buffer does not have protections against overfilling the buffer.
The term “stack smashing” refers to a specific attack that exploits a buffer overflow. This
attack is effective against programs written in modern program languages that use the stack
structure in invoking programs and subroutines. Whenever a program or subroutine is
invoked, the run-time system builds a stack including both data and return addresses. The
goal of stack smashing is to overflow a data buffer in the stack and alter a return address.
One should ask why buffer overflows are so common. There are a number of reasons, but
the main factor seems to be an obsession with program efficiency. The code required to
insure against buffer overflows makes the array operations a bit slower. To those of us who
are dedicated to quality software, this is a bit like taking the lifeboats off the cruise liner so
that it can make better time. Perhaps the author of these notes is too cynical.
The student should be aware that many of the programming languages contain constructs that
allow buffer overflow if used carelessly. Modern programming languages, such as Ada™,
that automatically generate code to prevent buffer overflows, have not been widely accepted.
The C and C++ programming languages have a large number of features that will allow
buffer overflow. The following table is adapted from the figure on pages 152 and 153 of the
book Building Secure Software by John Viega and Gary McGraw published by AddisonWesley in 2002. The ISBN is 0201-72152-X.
Function
gets
strcpy
strcat
sprintf
scanf
sscanf
fscanf
vfscanf
vsprintf
vscanf
vsscanf
streadd
strecpy
strtrns
realpath
syslog
getopt
getpass
Severity
Most risky
Very risky
Very risky
Very risky
Very risky
Very risky
Very risky
Very risky
Very risky
Very risky
Very risky
Very risky
Very risky
Risky
Very risky
Very risky
Very risky
Very risky
Comments (from the book by Viega & McGraw)
Use fgets (buf, size, stdin). This is almost always a big problem.
Use strncpy instead.
Use strncat instead.
Use snprintf instead or use precision specifiers.
Use precision specifiers or do your own parsing.
Use precision specifiers or do your own parsing.
Use precision specifiers or do your own parsing.
Use precision specifiers or do your own parsing.
Use vsnprintf instead or use precision specifiers.
Use precision specifiers or do your own parsing.
Use precision specifiers or do your own parsing.
Make the destination at least four times the size of the source.
Make the destination at least four times the size of the source.
Check that the destination is at least as big as the source string.
Manually check the size of the input arguments.
Truncate all string inputs to a reasonable size.
Truncate all string inputs to a reasonable size.
Truncate all string inputs to a reasonable size.
The author of these notes has become paranoid about string handling in the C and C++
programming languages. These languages use “null terminated” strings rather than storing
the length of the string explicitly. Failure to terminate the string properly can lead to large
chunks of the program code being appended to the string, with possibly amusing results. The
following code fragment, taken from the book referenced above, shows preferred practice.
// Copy into locations 0 through (dst_size – 2) only.
strncpy( destination, source, (dst_size – 1) ) ;
// Force the last character to be null.
destination[dst_size – 1] = ‘\0’ ;
The author of these notes is guilty of many buffer overflows in his programming, most often
in code written in FORTRAN, a language specialized for scientific processing. Consider the
following code fragment.
SUBROUTINE BUSTIT
DIMENSION A(10)
DO 20, J = 1, 80
A(J) = 0.0
20 CONTINUE
RETURN
END
One does not need to be a FORTRAN guru to note that the array is being overfilled. The
behavior of this code depends on the fact that early FORTRAN compilers allocated memory
statically, with code and data interspersed. The effect of this error is to overwrite a big block
of executable code, which when encountered will immediately terminate the program.
Other non-malicious code problems
The book discusses two other non-malicious code problems: incomplete mediation and
TOCTTOU. Incomplete mediation seems to be a fancy name for not verifying that user input
is valid for the problem. One key rule of software engineering is never to trust user input.
TOCTTOU (or TOCTOU) stands for “time-of-check to time-of-use” and refers to using
input that has been checked but changed since the time of its being checked.
Inadequate documentation can be considered another non-malicious code problem. This
problem arises when the limitations or assumptions of the code are not documented and made
obvious to future developers. This might as well be a problem with the code itself.
Malicious Code
Malicious code probably would not exist if it were not for non-malicious code errors. This
may be a simplistic view, but it does reflect the fact that much of the existing malware (a new
term for malicious code) is designed to take advantage of non-malicious errors.
The two main types of malware are viruses and worms. A virus is a fragment of code that
attaches itself to a running program, “infects” that program, and uses that program to “infect”
other programs. Viruses used to propagate via swapping disk drives, but now propagate
mostly via the internet. A worm is stand-alone program that propagates itself without the
help of another program. Worms generally propagate through the internet.
As an aside, we mention that the safety precautions once suggested to prevent spread of
computer viruses by swapping disks sounded very much like the suggestions to avoid spread
of sexually transmitted disease in humans, leading to the slogan “safe hex” – where the
reference is to “hexadecimal”.
There are other classifications of malicious code, including Trojan horses, logic bombs,
trapdoors, and rabbits. A Trojan Horse is a program that contains unexpected malicious
code that possibly operates in a stealthy mode. Many Trojan horses are spread by worms, but
this is not necessary. One of the earliest Trojan horses was described in a paper by Ken
Thompson in 1983, where he described a way to create a compiler that would infect the
system login script with malicious code to allow unauthorized logins.
A logic bomb is a program that is triggered when some event occurs. A time bomb is a
logic bomb that is triggered by a specific date and time. One of the more famous time bombs
was detected before it could be triggered on the anniversary of the creation of the state of
Israel. A trapdoor is an undocumented feature in a program or operating system that allows
certain persons (usually the system developers) to gain access independently of the normal
security system. These trapdoors are often inserted for maintenance or diagnostic purposes,
but can be discovered by malicious hackers and misused in any number of ways.
The term “rabbit” seems to have been coined in 1989, but is not much used. This malware,
which operates by replicating with the goal of exhausting some computer resource, has also
been called a “bacterium”, another term not much used any more.
More On Viruses
The textbook presents a good discussion of viruses, focusing on a number of important
topics. One of the more important topics is that of virus signatures and virus scanners.
Many computer viruses have sequences of bytes that can be used to identify them. It is these
sequences that are the targets for the virus scanners. Of course, a virus scanner can detect
only those viruses for which it has patterns – so keep your scanner up to date.
Polymorphic viruses are an attempt to deceive virus scanners by presenting the same virus
with a different signature. Most scanners will be updated as soon as a new variant of an
existing virus is detected, so again – keep your scanner up to date.
There is a new class of anti-virus software that the author of these notes has seen advertised,
this is a behavior-based scanner. Rather than having a list of virus signatures, the scanner
has a list of dubious behaviors, such as writing to the boot sector of a hard disk or attempting
to change certain system files. The user is allowed to define a list of “dubious actions”, each
of which will be trapped and not allowed without an additional user input. This class of antivirus software appears promising, but should be used with the pattern-based scanners as a
part of a defense-in-depth strategy.
Cryptographic hash sums are another tool for detecting virus infections on a disk. These
sums are the output of cryptographic hash functions that we have already seen in Chapter 2
of the textbook. For those who ask, a cryptographic hash function is just a hash function that
is of high-enough quality not to be broken easily by standard cryptanalysis techniques. One
identifies a file or set of files, performs a hash sum of these files, stores that hash sum in a
secure place (perhaps off-line), and periodically checks the hash of the files to see if
something has changed. In this case, as in many others, a change does not automatically
mean that something bad has happened, just that the change needs to be explained.
The idea of defense in depth is that one has multiple layers of protection against malicious
attacks, so that if one layer is breached there is a chance that the attack may still be stopped.
One obvious part of such a strategy is that each breach be detected and remedied as soon as
possible, so that one is not depending on one strong link in a weak chain for protection. The
idea of defense in depth comes from military strategy, where one does not want a single
penetration by the enemy to lead to a complete rout.
Defense in depth is best illustrated by cases in which it was not applied. One famous case
comes from the battle of New York, which began on August 27, 1776. General Washington
had arrayed his troops in a well-fortified line and was awaiting an attack by the British, who
seemed content to make a big display but not to advance. Unknown to the Americans, the
British had sent ten thousand men and fourteen fieldpieces on a circuitous march along the
Jamaica road that lead behind the American lines. Apparently, Washington thought that the
road would be patrolled, but nobody actually implemented that plan. While the British were
keeping his attention to the front of his lines, they were attacking the rear in a “sandwich
attack”, rarely attempted because few commanders were foolish enough to allow it to
succeed. Washington barely escaped with his army; New York City fell to the British on
September 16, 1776 and was occupied for the duration of the war.
More on Worms
The textbook then discusses two of the more famous worms from the recent past: the Morris
worm of November 2, 1988 and the Code Red worm from 2001. Each seems to have been
written just to propagate itself and not to attack any systems directly. Should the worm have
carried a malicious “payload” (code to attack an infected system), either could have done
much greater harm. As it was, each achieved its effect by spreading rapidly and consuming
internet resources. The most embarrassing fact related to the Code Red worm is that it
exploited a widely-known vulnerability for which a patch had been available for some time.
Trapdoors
The term “trapdoor” has several uses in information assurance. Within the context of
cryptography, it may refer to a trapdoor function – one that is invertible and easily
computer but for which the inverse function is very hard to compute. Within the context of
this discussion, a trapdoor is hidden (and often undocumented) code that can be invoked by
specific calls that also are not documented. Trapdoors in this context are often remnants of
the software testing process, in that they were used to allow the developers to remove errors.
These remnants are usually left in the code by mistake.
Two of the more famous trapdoors are the VAX/VMS special accounts and the Unix
sendmail trapdoor. A VAX computer, freshly delivered from DEC (the Digital Equipment
Corporation – now defunct) was configured with the VMS operating system having three
privileged accounts – SYSADMIN, FIELD, and one other that I forget. Each of these
accounts had “superuser” privileges (DEC did not actually use that term) in that a process
running on the account could access and modify any of the computer’s resources, including
all of the files. The fixed password for the FIELD account was “SERVICE”. Each of the
other two accounts also had pre-defined passwords. The expectation was that each of these
passwords would be changed very early in the system generation process. In fact the
SYSADMIN password was often changed but the other two accounts were ignored. Thus, a
clever malicious hacker had two well-known trapdoors into the system. That which
facilitates system maintenance also facilitates hacking.
The author of these notes is still doing some research on the Sendmail trapdoor. It is
obvious that the trapdoor was a feature added by the developers with the goal of facilitating
the debugging of the sendmail code. The sendmail facility was created in order to allow
remote users to send and receive e-mail. The trapdoor refers to the fact that a remote user
could place sendmail in a debug mode (some documents refer to this as “wizard mode” in
distinction to the usual debug mode) and gain root access to the system running the program.
While this greatly facilitated correcting any program errors, it certainly opened the door wide
for malicious hackers. Recall that a person or process running with root access on a UNIX
system can access and modify any system resource.
The Salami Attack
The book discusses this attack method in the context of banking, where it is most often seen.
Steal a penny from enough people and you may have accumulated a lot of money.
Covert Channels
The textbook presents a covert channel in terms of transmitting data at a small, but
significant, rate. To make the covert channel threat appear a bit more real, I present the
following scenario in which a two-bit message can be worth millions of dollars.
Suppose I am a staffer in the FDA (U.S. Food & Drug Administration) who is involved in
examining a cancer drug for possible approval. Just for fun, let’s imagine the company
developing the drug is called “ImClone” (now where did I get that name?). Suppose that I
have a friend, called “Martha” (the name is fictitious, picked purely at random – I assure
you), who is heavily invested in ImClone stock and who would certainly appreciate a tip
from me before the approval or lack of approval is announced publicly.
Martha’s investment choices might be based on prior knowledge of the announcement. The
stock has already gained significant value based on the preliminary announcement of the
cancer drug and the fact that it has been submitted for FDA approval. Should the drug be
approved, the stock value will go up; in that case Martha would want to buy more stock.
Should the drug not be approved, the stock value will decrease significantly; in that case
Martha would want to sell the stock at the current high price and possibly repurchase the
stock after its value had fallen. My passing such confidential information to Martha before it
is made public, and her acting on it, is an illegal practice called “insider trading”.
Here is the covert channel for communicating the decision. I create two text files, call them
zonk,txt and bonk.txt (good luck on finding any significance in those names) and apply file
locking with the following agreed interpretation. The files are assumed to contain textual
information that is completely innocuous and not likely to arouse suspicion.
zonk.txt locked?
No
No
Yes
Yes
bonk.txt locked?
No
Yes
No
Yes
Interpretation
The FDA has not yet decided
on the approval of the drug.
The FDA will not approve the drug.
The FDA will approve the drug.
Now my friend Martha can agree to check these files often and get enough warning to profit
illegally based on inside information. We assume that Martha is otherwise squeaky clean
regarding this transaction, so that the proof of insider information would be quite difficult.
The textbook discusses other ways of covert transmission of information. Some, such as
timing channels, might be more of a theoretical worry as random variations in system
timings may make such channels unreliable. The bottom line is that covert channels are very
hard to detect and almost impossible to prevent.
One might say that the best deterrent to covert channels is happy employees. We shall
discuss organizational security policies a bit later in the course. The bottom line here is
quite simple: everybody having access to the software must want it to be secure.
Software Engineering
The discipline of software engineering arose from the experience that people at IBM had in
the development of the OS/360 operating system for the new 360 line of computers. Some of
the lessons learned in that project were reported in the book The Mythical Man Month:
Essays on Software Engineering, by Frederick P. Brooks. The book was first published in
1975 and reprinted in 1995 by Addison-Wesley (ISBN 0-201-835959).
There have been a number of approaches to software engineering since its origin as what
probably would be called Structured Analysis / Structured Design (SA/SD) today. Other
approaches include data-flow analysis and object-oriented analysis and design. In this
author’s opinion, each approach to software engineering brings something to the design
process. The book discusses a few topics of importance.
The first design topic has negative overtones; it might be called the commercial imperative.
Most company managers want software delivered on time and under budget; these often have
little experience in the development of software. This commercial imperative often leads to
pressures to deliver a product despite a number of obvious problems. In this context, quality
software engineering may be seen as a hindrance that is barely to be endured. No company
will officially admit such practices, but the author of these notes has been there.
The lessons of structured analysis and design that remain valid today focus on modular
development of software. One idea is that there is a maximum size for each module, be it
subroutine or function. It used to be decreed that no module have length greater than 65
lines. This gave rise to numerous discussions – was this total lines or lines of code, and what
defined a line of code in a language (such as C++) allowing for multi-line expressions. After
some legalistic wrangling, it was decided to allow the maximum number to be a bit vague.
In this author’s opinion, 500 lines of code in a single module is probably excessive and 25 to
50 lines may be a bit small.
Two other ideas from SA/SD were functional cohesion and module coupling. Functional
cohesion measured the “focus” of a module – how clearly defined its goal was. Thus a
function to calculate the sine of an angle (and nothing else) would rate as high cohesion,
while a module containing the sine, cosine, and tangent functions would rate only a bit lower
in cohesion, but still acceptably high. Add a square root function to that module and the
cohesion becomes questionable, then add a sort routine and the cohesion would reach the
lowest value of “coincidental” – just a bunch of code with no specific purpose.
Modular coupling is a measure of how one module depends on the implementation of
another module. The desired goal is loose coupling, in which modules interact only by
argument lists. Modules that share external storage are slightly more coupled (less desirable,
though this might be acceptable), while module that depend on the internal data structures of
other modules are unacceptably coupled. The goal here is to be able to change the
implementation of any module while preserving its interface to the rest of the system. This
goal has been adopted by object-oriented design methods, where it is called encapsulation, a
term that probably was not used in SA/SD.
One of the more interesting approaches to software engineering is called the spiral model,
although there are quite a few variants of this one. In order to understand this model, one
should understand the overly-optimistic waterfall model, in which software development
progresses in a number of well-defined steps.
1)
2)
3)
4)
5)
Requirements analysis and definition of system tests
System analysis
System design
System coding
System testing
The problem with this approach is that there are very few systems for which the requirements
are well enough understood in advance for the approach to be applied. Quite often, the
process of design and coding of a system will bring new system requirements to light. The
spiral model allows for this in that the steps of the waterfall model are undertaken numerous
times until a stable set of requirements is agreed upon.
Software testing is definitely an art in that no completely systematic approach may be
defined. There are a number of standard approaches that should be followed, but the student
should be aware that these are merely necessary and not sufficient. The first requirement
(and one of the least followed) is that a complete system test be specified at the same time as
the requirements. This suite of tests should be refined as the system requirements change and
be developed and available at the same time as first code on the system itself is delivered.
Too often it is the case that the tests are written after the system code is delivered, and quite
often they are carefully designed so that the system will pass – not a responsible idea.
Who should do the testing? It is obvious that each developer should do some testing,
focusing on errors that are easy to identify. It is at this state of development that most buffer
overflow errors should be identified and eliminated, for example. At the same time, there
must be an independent testing organization as a part of any group designing software of
even moderate complexity. This author worked for a company that developed software to be
used to manage hotel and motel properties. The development team included an independent
testing team, all members of which were drawn from the lodging industry. None of the
members of this testing team had experience in software development, but each knew what
the software had to do. This team developed a very good product, although a bit late.
An important aside here is that system security requirements are an essential part of the
system requirements. The system security must be completely understood before any code is
written or it is likely to be relegated to a side project, and security holes will abound.
Peer reviews (structured walk-throughs) are other tools that are rarely used. In these, a
developer presents his or her software design to the group, under the understanding that the
topic for review is what the module does and how well it adheres to the specification. These
reviews are supposed to be “egoless” in that developers not take it personally when the
software is criticized – fat chance of that.
Configuration Management and Associated Testing
One of the major concerns in the development of major software projects will come as a
complete surprise to a programmer who has worked alone during college years. This is the
management of changes made by a number of people.
One of the better tools is Visual Source-Safe, marketed by Microsoft. This is a repository of
documents (code and text documents) that allows check-out by individual developers. This
enforced serialization of software changes should prevent proliferation of multiple copies of
the software. This author has been a part of two teams using Visual Source Safe; it is only a
minor nuisance and it does seem to do the job.
Two important types of tests are regression testing and what I call targeted testing.
Regression testing is a by-product of software testing and occurs in the following context.
When a software error is discovered in the normal process of testing, it is fixed and should
never occur again. To insure that it never occurs again, a test specific to the fault just
discovered is created and entered into the test suite. The modified software is run through
the modified test suite to insure that the error has indeed been fixed and that no previous
errors reappear due to problems in writing software patches.
Note that regression testing targets only those errors that have been discovered. The
targeted test is based on a developer’s understanding of what might go wrong in software
and devising tests based only on intuition. The author of these notes, being tasked with
writing tests for a software system, made just an assumption and created a valid data input
that caused the system to crash. This error was acknowledged with some grumbling and
fixed. Never think that being a proficient tester will make you popular.
We end our notes on this chapter with a big dose of cynicism. Proofs of program
correctness are impossible and most often a waste of time. Certain errors, such as buffer
overflow and incorrect input, can and should be anticipated. The program should be
considered logically, apart from “debugging”, and visually scanned in an attempt to
demonstrate correctness. If you have the courage, have another developer read your code.
This can be an enlightening and often humorous experience.
Final rule – do not give in to the pessimism that it is impossible to create error-free software.
While it probably is impossible, we should keep at it.