Runtime Code Modification for Avoidance of License Server Authentication
Gregory Deych and Gokul Nadathur
Computer Sciences Department
University of Wisconsin – Madison
{gdeych,gokul}@cs.wisc.edu
December 6, 1999
A general strategy for avoiding authentication by license servers has been developed. The strategy involves the
modification of the execution path of the licensing process at run time without the process recognizing that such a
modification has taken place. The necessary tools for such an exercise were created using the Proc file system interface
provided by Solaris OS. These tools have the following function: stopping a process at call points to system calls,
forcing a process to trace certain signals and faults and modifying the address space of the process at runtime. The
strategy was tested on Adobe Systems Framemaker and the licensing process was subverted.
1
Introduction
This paper chronicles our investigation of the license server – client interaction. Our project concentrates on
bypassing or removing the elements of licensing that relied on communication with the license server. It is
possible that some aspects of licensing depend on other methods, in which case they are considered to be outside
our purview for the purposes of this project.
Our strategy relies on steering the execution of the program at runtime. We do not make any changes to the
static object code, which means that our mutator must be executed every time the program is run. The strategy
was tested on Adobe Systems Framemaker. We launch two processes: a clean Framemaker process, with
licensing code intact and a Code Changing Tool that will create all the modifications to the Framemaker process.
The tool process terminates itself after modifications have been carried out, leaving only the Framemaker
process with network licensing functions disabled or bypassed.
1
We begin by describing the general licensing server paradigm in Section 2. In Section 3, we describe our
strategy for avoiding the licensing process. In Section 4, we discuss the tools we have built to implement our
strategy. In section 5, we describe our experiments on Framemaker as well as other programs. We finish the
paper by describing additional enhancements that could be added to our tool set to achieve additional
functionality, as well as how application makers may safeguard their programs from the techniques we have
utilized.
Framemaker
Launching
Process
Modified Framemaker
Code Changing Tool
Figure 1: Launching Process creates both of the processes. The command invokes both the tool and
‘maker’.
2
Mechanism behind using a License Server
A license server is supposed to keep applications from running, unless it is satisfied that the user requesting the
application possesses a proper permission to run that application. This permission may be a personal license for
Figure 2: Conceptual view of the major functional blocks of the typical
license server. The server interface may be embedded in or “wrapped ”
around an application.
Permission
Distribution
Access Request
Application Release
Permission to Run
Access
Control
Server I/F
(Developed by vendor)
License
Database
License Administration
2
that particular user or a site license specifying a number of applications that can be run at the same time. Various
license servers implement a particular license-managing model, or implement several to be chosen at the
convenience of the site administrator. The application programmer may develop the license server together with
the client application, or the server may be an external product integrated with the application. A diagram of a
license server operation is depicted in Figure 2. The application submits a request to be executed to the license
server, together with some identifying information. The particulars of the information depend on the license in
question, but commonly include the user name and location of the machine on which the request is to be
executed. The license server evaluates the request, records it in the license database and sends the release to the
application. Upon completion, the application typically notifies the server that its use of the license is finished.
3
General Strategy for Avoiding the License Checking
The client application must contact the license server to determine if it is in fact licensed. If it is unable to do so,
it defaults into unlicensed mode. Our general approach to the problem is two-fold: to modify the runtime process
to steer the execution path around the licensing checks and correct any error flags generated due to such a
modification. While the communication with license server will not affect our approach, its absence validates
our result. Since we had decided to do all of our modifications dynamically, we needed a set of tools that would
allow us to modify the process in the memory, as it is loaded and executing.
By stopping the process and disassembling its functions, we can obtain information vital to our strategy. First,
we need to define a few terms:

Trace point – the virtual address of the instruction that would need to be modified, as well as what
instruction would need to overwrite this address.

Trace of a process - the set of all trace points of a particular process
Our task is to analyze the runtime binary in order to determine its trace. An intuitive step would be to stop the
process when it executes a connect call. The process can then be loaded into a debugger, and the process stack
examined. This strategy enables us to build a partial process call graph, constructed from the bottom up. Once
3
we have created this graph, we are ready to examine the program’s behavior as we mutated the graph by pruning
some of its links.
Figure 3: General Scheme of Control Flow Mutation
Successful
Licensing
Successful
Licensing
Path Executed
Main
Main
Path Not Executed
License
License
Net License
connect
Net License
License Error
Mutated
connect
Mutated
License Error
success
Exit
failure
Exit
Embedded
Traces
Our mutation strategy employed one or more of the following schemes:

Remove a particular function from the execution tree.

Eliminate the detection of the removal of the function.

Branch to points that would be executed if the licensing was carried out successfully.
If our mutation of the trees caused an error condition, the exit points would lead us to the location of the error.
Once the location of the error is uncovered, we either bypass the error detection code or disabled the error exit
4
code, which prevented the process from properly responding to the lack of licensing information. By applying
this method, we are able to disable a large portion of the licensing functionality.
4
4.1
Tools
GDB debugger
We use GDB primarily as a disassemble tool, breakpointing on an entry to a function we want to investigate and
disassembling it. This allows us to see the assembly code of the function, which gives us some insight into its
function as well as strategies for its modification. Unfortunately, due to the lack of debugging information in the
program binary, use of GDB as a debugger is impossible.
4.2
Tools Using Proc File System
We had to create a number of tools to allow us to implement the changes to the runtime program code. For this,
we have chosen to use the Proc file system, which is used to access the state of the process in the Solaris
operating system.
/proc - the process file system
/proc is a file system that provides access to the state of each process and light-weight process (lwp) in the
system. The name of each entry in the /proc directory is a decimal number corresponding to a process-ID. These
entries are themselves subdirectories. Additional files contained within each subdirectory provide access to
Process State. Standard system calls like open, close, read and write are used to access the files in the /proc
directory. The following is the description of files in the /proc directory that are relevant for creating the tools.
Structure of the /proc/pid (pid - process id)
as -This file contains the address-space image of the process. It can be opened for both reading and writing.
ctl -A write-only file to which control messages can be written, directing the system to change some aspect of
the process’ state or to control its behavior in some way. These control messages include stopping a process at
the entry and exit points of specific system calls and forcing the process to trace certain signals and faults.
5
status- The status file contains state information about the process and the main thread of the process. When the
behavior of the process changes in certain predefined ways such an execution of a system call, reception of
signals or when the state of the process changes, appropriate information is written to the status file.
Tools created using /Proc:
The tools created using the Proc file system had the following functions.
1. A process could be stopped at the entry or exit point of a system call by writing the control messages
PCSENTRY/PCSEXIT to the ctl file of the process. When it is stopped at the entry to the system call, the
parameters have already been loaded. The values of these parameters are stored in specific fields in the status
file. So for every particular system call that the process emits it is possible to examine the values it passes. This
can be used to get information about the domain (TCP or UDP) of its socket calls, information about the server it
connects to and the byte sequence it sends and receives.
2. A process could be made to trace certain signals and faults and stop when such signals or faults occur. The
signals or faults might be triggered by certain operations inherent to the process or due to certain control
messages written to its ctl file by another process.
3. Modifying the Address space of the process:
(a) The address space of the process can be modified by writing to the as file in the /proc directory of
the process. We synthesized machine instructions for branching (conditional or unconditional), no
– op instructions and for moving data between registers. After these instructions have been
synthesized, the address space was modified at certain addresses as dictated by the trace points. By
modifying those addresses, we were able to steer the execution of the program according to our
strategy.
(b) To examine the results of certain conditions or the return values of certain functions, temporary
modifications were made to the address space in the form of breakpoints at arbitrary instruction
addresses. This is beyond the capacity of GDB, since it recognizes only points like function entry
or exits if debugging information is not built into the executable code of the process. After a
6
breakpoint has been set, there are two more steps to be done. The breakpointing process has to be
forced to trace the relevant faults, which can be done using the tools in step (2). Secondly, there has
to be a mechanism for the external tool to identify whether the process that it’s monitoring has
faulted on the breakpoint or not. This can be accomplished by polling on the file descriptor of the
status file. When a fault occurs, the reason for the occurrence is written to the status file. By
polling the status file for normal or high priority write operation, the tool recognizes the execution
of the breakpoint instruction. After the process stops on the breakpoint instruction and has been
examined, the original instruction (replaced by the breakpoint) is written back and the process is
continued.
4. Watching the process:
Another useful operation is a watch on the address space of the process. A section of the address space of a
process can be watched for read, write or execute operations. The watch operation forces the process to fault
when it does the specified activity on the watched area of the address space. By forcing the process to trace the
fault, the process can be stopped and its state can be examined. This is useful is when the address of the data
where some manipulation would take place is known, but not the instruction that would perform the
manipulation. When a process receives data from the remote server, the buffer in which the process places the
data can be determined by examining the parameters of the recvfrom() call. The points in the code of the process
that would manipulate this buffer can be found out by watching this buffer for a read or a write access. The point
can then be used as the root of the control flow tree that essentially traces how the licensing is done.
5. Monitoring multiple processes:
Another interesting problem lies in developing mechanisms to monitor multiple processes. This is useful when
licensing is done in one of the child process of the process that is being monitored. An example of this situation
is cited in section 5.2. The Proc file system offers a simple and elegant solution to this case. The idea is to setup
the proc state for which the child should respond in the parent process itself and turn on the "inherit on fork
mode" in the parent process. This causes the child to inherit the proc state of its parent including the signals,
faults and system calls it should trace. However, it should be noted that watch areas are not inherited.
7
5
Applying the Strategy
5.1
Framemaker
Adobe Systems’ Framemaker is a desktop publishing application for the UNIX environment. It utilizes FLEXlm
license server to control access to itself. Framemaker is started by invoking a series of script files which set up
environmental variables, determine the machine architecture the client is to be run on, and execute an
appropriate binary.
Figure 4: Scheme of Framemaker subroutines
Main
Legend
UiInitPrinter (Intialize printer servers)
Function call
Erased
Mutated
NlInit
NlGetLicense
FlcToFlsCheckOut
connect
NluiCheckOutLicense
Mutated
Open file
makerMainLoop
Open new file
CheckOutLicense
NlGetLicense
Exit from maker
8
During startup Framemaker executes a number of functions related to general initialization of its state, as well as
some directly related to the licensing process. The functions named Flm* control the general licensing process.
Other functions, of type Nl* control network licensing and communication with the license server. In fact, much
of the licensing is accomplished through the function NlInit, executed from the main routine. NlInit performs
license checking on startup of Framemaker. Another license check is performed when a user opens a file. In
addition, the client application contacts the server at the end of the user session to notify the server that the
license has been released. By applying our strategy, all contact between license server and Framemaker is
eliminated.
The flowchart (figure 4) depicts the structure of the main() function of the Framemaker. The first branch is the
connection to the printer server. The connect calls performed in the NlInit are the primary connections to the
license server. As we have found out, if those are disabled Framemaker tries to perform the licensing one more
time on Open File and New File commands. In addition, it also contacts the server on Exit. As pointed out in
the flowchart all the licensing calls in these cases flow through the function CheckOutLicense(). The actual
licensing was done NlGetLicense() that returned a value 1 on success and 0 on failure.
The mutation done to CheckOutLicense() is as follows:

Erase call to NlGetLicense .

Change the register where the return value was stored to one.
Identifying that the license mechanism has been bypassed
PID
Call
Type
Phase
23582:
23582:
23582:
23582:
connect(4, 0xEF4B1738, 16)
connect(3, 0xEFFFEE58, 16)
connect(5, 0xEF4B1738, 16)
connect(4, 0xEF4B1738, 16)
Maker: Starting FrameMaker 5.5.6.
Copyright (c) 1986-1998 Adobe Systems
Incorporated
connect(4, 0xEF4B1738, 16)
connect(10, 0xEF4B1738, 16)
Xserver
Xserver
License Server
License Server
Startup
Startup
Startup
Startup
Startup complete
Printer
License Server
After Startup
Exit
23582:
23582:
Table 1: Unmodified Framemaker connect calls
9
Two ways were used to verify our traces.
1.
The connect calls emitted by the process and the points at which they were emitted was traced. After
applying our trace to the Framemaker, we eliminated all connect calls which were not related to licensing,
while preserving the XServer and Print Server connection. The results are presented in Table 2.
PID
18322:
18322:
18322:
Call
connect(4, 0xEF4B1738, 16)
connect(3, 0xEFFFEE58, 16)
Maker: Starting FrameMaker 5.5.6.
Copyright (c) 1986-1998 Adobe
Systems Incorporated.
connect(5, 0xEF4B1738, 16)
Type
Xserver
Xserver
Phase
Startup
Startup
Startup complete
Printer Server
After Startup
Table 2: Modified Framemaker connect calls
1.
We have further tested our theory by supplying an incorrect license server to Framemaker. This was done by
removing the initialization to the Environment variable FM_FLS_HOST . This initialization was done
statically in a script file that was executed before the actual Framemaker executable was overlaid on the
process space. The default license server was named stimpy. When Framemaker was run using the modified
script, it defaults to demo mode giving a message that license was not installed. When our traces were
applied, Framemaker executed with full functionality.
5.2 Experiences with Purify
This section illustrates a situation different from one in Framemaker and cites mechanisms for dealing with it.
Purify, a runtime error detection tool from Rational Software, also uses a server to perform license
authentication. Since Purify itself launches several processes, the problem of multiple processes being involved
is inherent and cannot be avoided. During the preliminary experiments we conducted using Dyninst (yet another
powerful tool for parallel debugging), we found that Purify spawns two processes. One is the process it monitors
and the other is a process called rtslave. The process rtslave verifies license and hence has to be monitored by
Dyninst. Dyninst did not have support for monitoring multiple processes, so we had to write a tool using Proc
file system that linked Dyninst with rtslave. The problem could be solved by setting up the proc state for rtslave
10
in the parent process and force rtslave to inherit the proc state. In this way, rtslave can be made to stop when it
tries to execute a connect call. Afterwards, the strategy used in Framemaker can be applied. We would like to
emphasize that this section has been written merely to illustrate the use of our tools in handling multiple
processes. We have not conducted thorough experiments on Purify and the question of avoiding licensing in
Purify remains open.
6
Ways to defeat our scheme
There are a number of ways in which a developer can hinder our scheme:

Since we rely on the access to the Proc file system, if the process locks its own directory in the Proc file
system our approach will not be available

Likewise, should the process use the setuid program to raise its privileges, we may have difficulty altering
its process space. However, this is not a big problem, since we can stop the process after it has execed its
image and prevents the setuid call.

The license checking can employ some functions on a probabilistic basis, executing some elements of
license checking only occasionally. This will increase the amount of time needed to profile the control flow
of the license checking mechanism, since not all functions will be executing every time.

Another measure that may delay our progress is naming all functions with names not indicative of their
function. In Framemaker, we were able to quickly zero in on routines performing licensing since their
names clearly indicated their functions.

The most promising approach appears to be executing critical portions of the client application via an RPC
call to the license server. Since our approach relies on removing the communication links between the
server and the client, this would preclude the application from executing. However, this has effect of
increasing the necessary server-client communication, with the concomitant result of increasing start up
time.

An admittedly extreme approach would be to add hardware encryption on the memory level. If we cannot
read the memory state of the process, we would not be able to get any information about how to modify it.
Of course, this involves additional cost and inconvenience, and is unlikely to be used for a consumer
application.
11
7
Conclusion
Our strategy is applicable to many applications that use license servers as means of authenticating their license
status. In the case of Framemaker, by strategically removing branches from the execution tree, we were able to
fool the program into believing that the absence of communication from the server is the correct state of affairs.
We have also cited mechanisms for overcoming difficulties posed by multiple processes, although thorough
experiments still need to be done in such cases to give a final judgement on whether our strategy is truly
universal or not.
12
Reference
1.
Conceptual Description of a Generic License Manager. By Jim Geisman, Marketshare, Inc.(
http://www.globetrotter.com/ms_lm.htm)
2.
DyninstAPI Programmer’s Guide
(http://www.cs.umd.edu/projects/dyninstAPI/rel1.2/dyninstProgGuide.pdf)
3.
Unix Programmer’s Manual (/common/usr/man/vendor/sun4x_56/man4/proc4)
4.
Sparc Assembly Language Reference Manual (http://docs.sun.com/)
13