InspectOS - Extending Features for
Operating System Inspection
Supervisors:
Drd. Ing. Octavian Purdilă
As. Drd. Răzvan Deaconescu
Bachelor Thesis
Computer Science and Engineering Department
Automatic Control and Computers Faculty
University Politehnica of Bucharest
July 2011
Author:
Alexandru Șutîi
Abstract
Operating Systems are one of the most interesting Computer Science fields. In order to become an
excellent software engineer, one should have at least basic knowledge about the operating systems
concepts. However, for most of the students it is hard to understand many of them.
The most spread teaching technique is with repeated explanations of the low level mechanisms. As
there are some tools that inspect the processes for industrial purposes, there is a lack of educational
tools that would present the OS concepts dynamically to the students based on a given program.
InspectOS is coming to solve this problem. It is a useful educational tool that gathers information
about a traced process directly from the running operating system. It comes to extend the OS-Tool,
an application developed a year before by our colleagues Lucian Cojocar, Vlad Dogaru and Andrei
Buhaiu.
In this paper we shall present both the initial and the added features, however focusing more on the
design, the implementation and the testing of the newly added features.
Contents
InspectOS - Extending Features for Operating System Inspection ......................................................... 1
Abstract................................................................................................................................................... 2
Acronyms ................................................................................................................................................ 5
1
Introduction .................................................................................................................................... 6
1.1
InspectOS ................................................................................................................................ 6
2
Operating System Theory ............................................................................................................... 7
3
Architecture and Features .............................................................................................................. 9
4
3.1
Architecture ............................................................................................................................ 9
3.2
OS-Tool Features .................................................................................................................. 10
3.3
InspectOS New Features ....................................................................................................... 11
Implementation ............................................................................................................................ 12
4.1
Underlying Technology ......................................................................................................... 12
4.1.1
GTK ................................................................................................................................ 12
4.1.2
Graphviz and Cairo........................................................................................................ 12
4.1.3
GDB ............................................................................................................................... 12
4.2
Graphical User Interface Redesign ....................................................................................... 13
4.3
GDB Asynchronous Commands ............................................................................................ 16
4.3.1
Another Possible Implementation ................................................................................ 17
4.3.2
A Known Issue ............................................................................................................... 17
4.4
Thread Support ..................................................................................................................... 17
4.4.1
4.5
Process Standard Input ......................................................................................................... 19
4.5.1
4.6
Known Issues ................................................................................................................ 20
Process Standard Output ...................................................................................................... 20
4.6.1
The First Failed Approach – Redirect the Process Output ............................................ 20
4.6.2
The Second Failed Approach – Parse the GDB Output ................................................. 20
4.6.3
The Final Solution ......................................................................................................... 20
4.7
5
Problems encountered ................................................................................................. 19
Variable Value Display .......................................................................................................... 22
Test Scenarios ............................................................................................................................... 23
5.1
Compiling and Tracing a Process .......................................................................................... 23
5.2
Unit Tests .............................................................................................................................. 23
5.2.1
Memory Management .................................................................................................. 23
6
7
5.2.2
File Descriptor Table ..................................................................................................... 24
5.2.3
Asynchronous Commands ............................................................................................ 25
5.2.4
Thread Support ............................................................................................................. 25
5.2.5
Process Output ............................................................................................................. 28
5.2.6
Process Input ................................................................................................................ 28
User Feedback .............................................................................................................................. 29
6.1
InspectOS Developers’ Feedback.......................................................................................... 29
6.2
Feedback from the OS-Tool Developers and our Supervisors .............................................. 29
6.3
Feedback from Students ....................................................................................................... 30
Conclusions and Future Plans ....................................................................................................... 31
A. Compiling, Installing and Running InspectOS ................................................................................... 32
System Requirements ....................................................................................................................... 32
Installation ........................................................................................................................................ 32
Cloning the repository .................................................................................................................. 32
Installing auxiliary packages ......................................................................................................... 32
Running InspectOS ............................................................................................................................ 32
Tracing a process .............................................................................................................................. 33
B. InspectOS Repository Description .................................................................................................... 34
References ............................................................................................................................................ 35
Acronyms
OS – Operating System
COW – Copy-On-Write
VMA – Virtual Memory Area
FDT – File Descriptor Table
GDB – GNU Debugger
GUI – Graphical User Interface
PID – Process Identifier
TID – Thread Identifier
Alexandru Șutîi
1 Introduction
An Operating System is a set of software programs and data that runs on a computer, manages the
computer hardware and provides common services for the application software [1]. The main goal
of an operating system is the hardware abstraction for the running processes. In order to do this an
operating system has a series of management mechanisms for every hardware resource type.
The main features of an operating system are:






Process management
Memory management
File system management
Networking
I/O
Security
In this paper I am presenting the InspectOS, a tool that provides dynamic information about some of
the most important above mentioned mechanisms: process management, memory management,
file system management.
1.1 InspectOS
InspectOS is a continuation of the project called OS-Tool developed by our colleagues Lucian
Cojocar, Vlad Dogaru and Andrei Buhaiu in 2010. They have designed the basic structure and worked
hard on the information gathering, processing and on presentation. They have done a great work,
but there were needed a series of improvements. Therefore we, Alexandru Șutîi and Radu Velea,
continued the project development, calling it InspectOS this year. As it is described later in the
Architecture and Features and Implementation chapters we have added a series of features and
performed some refactoring and redesign in the user space modules.
InspectOS – Extending Features for Operating System Inspection
2 Operating System Theory
In order to better understand the way InspectOS works and the concepts it is describing you have to
be presented some basic operating systems theory. I will present the concepts, their definition and a
short explanation.




Process - an instance of a computer program that is being executed. It contains the program
code and its current activity. Depending on the operating system (OS), a process may be
made up of multiple threads of execution that execute instructions concurrently [3]. A
process has an independent address space, a file descriptor table, a process id, etc.
Thread - In computer science, a thread of execution is the smallest unit of processing that
can be scheduled by an operating system. It generally results from a fork of a computer
program into two or more concurrently running tasks. The implementation of threads and
processes differs from one operating system to another, but in most cases, a thread is
contained inside a process. Multiple threads can exist within the same process and share
resources such as memory, while different processes do not share these resources. [4]
Differences between a process and a thread - Threads differ from traditional multitasking
operating system processes in that [4]:
o Processes are typically independent, while threads exist as subsets of a process
o Processes carry considerable state information, whereas multiple threads within a
process share state as well as memory and other resources
o Processes have separate address spaces, whereas threads share their address space
o Processes interact only through system-provided inter-process communication
mechanisms.
o Context switching between threads in the same process is typically faster than
context switching between processes.
Virtual memory - virtual memory is a memory management technique developed for
multitasking kernels. This technique virtualizes a computer architecture's various
hardware memory devices (such as RAM modules and disk storage drives), allowing
a program to be designed as though [5]:
o


There is only one hardware memory device and this "virtual" device acts like a RAM
module.
o The program has, by default, sole access to this virtual RAM module as the basis for
a contiguous working memory (an address space).
Demand paging - demand paging (as opposed to anticipatory paging) is an application of
virtual memory. In a system that uses demand paging, the operating system copies a disk
page into physical memory only if an attempt is made to access it (i.e., if a page fault
occurs). It follows that a process begins execution with none of its pages in physical
memory, and many page faults will occur until most of a process's working set of pages is
located in physical memory [8].
Copy-On-Write - Copy-on-write (sometimes referred to as "COW") is an optimization
strategy used in computer programming. The fundamental idea is that if multiple callers ask
for resources which are initially indistinguishable, they can all be given pointers to the same
resource. This function can be maintained until a caller tries to modify its "copy" of the
Alexandru Șutîi


resource, at which point a true private copy is created to prevent the changes becoming
visible to everyone else. All of this happens transparently to the callers. The primary
advantage is that if a caller never makes any modifications, no private copy need ever be
created. [9]
File Descriptor - In computer programming, a file descriptor is an abstract indicator for
accessing a file. Generally, a file descriptor is an index for an entry in a kernel-resident data
structure containing the details of all open files. In POSIX this data structure is called a file
descriptor table, and each process has its own file descriptor table. The user application
passes the abstract key to the kernel through a system call, and the kernel will access the
file on behalf of the application, based on the key. The application itself cannot read or write
the file descriptor table directly [6].
Virtual Memory Area - The virtual memory area (VMA) is the kernel data structure used to
manage distinct regions of a process's address space. A VMA represents a homogeneous
region in the virtual memory of a process: a contiguous range of virtual addresses that have
the same permission flags and are backed up by the same object (a file, say, or swap space).
It corresponds loosely to the concept of a "segment," although it is better described as "a
memory object with its own properties." [7].
InspectOS – Extending Features for Operating System Inspection
3 Architecture and Features
3.1 Architecture
InspectOS was designed with a high degree of modularity as can be seen in Figure 1. This approach is
better suited for parallel development and for efficient sustainment.
The four main modules are as follows:




The Gathering Module – This is the module that is responsible with gathering the
information about the traced process directly from the Operating System. It collects data
about the VMA structures, the File Descriptor Table and the Virtual and Physical Memory
pages. The data is presented to the user space via a procfs interface.
The Information Processing Module – As the GUI and Process Tracing modules, it is written
entirely in Python. The main purpose of this Processing Module is to take the information
from the Gathering Module via the procfs interface and transform in an easy presentable
way for the GUI module.
The GUI module – Displays the processed process information in a friendly way. Each
process is presented in its container with several panes displaying specific type of
information. The GUI also interacts with the Process Tracer for both sending commands to
the process and retrieving certain information about the process state.
Process Tracer – This is an interface for manipulating the process run state. It is a wrapper
over GDB and uses expect for interaction with GDB. It gives the opportunity to step the
process, provide process input and retrieve line information, process output and variable
values.
Alexandru Șutîi
Figure 1 - InspectOS Architecture Diagram
3.2 OS-Tool Features
As mentioned above OS-Tool is the application InspectOS began from. The architecture was
designed in OS-Tool and InspectOS came to improve the application with new features. The features
that OS-Tool already had are the following:




Virtual and Physical Pages – The application displays the table of virtual and physical pages
and show the correspondence between a virtual page and the physical frame it is mapped
on. Also it notifies whenever a Copy-On-Write or a Demand Paging event occurs.
File Descriptor Table – For the traced process you have the opportunity to visualize the open
file descriptors and the type of the open files.
VMA Tree – The red-black Virtual Memory Area Tree can be found in the VMA pane.
Source code visualization – Whenever a process is traced, its source code is displayed. By
pressing the Next button you can advance the process an instruction at a time.
InspectOS – Extending Features for Operating System Inspection
3.3 InspectOS New Features
InspectOS main purpose was to improve the graphical user interface in order to make it more
intuitive for beginners and extend the OS-Tool by developing new features.
Below you can see a list of new features we have developed:








GUI Redesign – In OS-Tool the GUI was divided in components displaying a specific resource.
For example, whenever a new process was created a new widget was added for each the
Source Code, Memory, File Table and VMA containers, rather than adding a container
encapsulating all above mentioned widgets. Therefore we redesign GUI so a better per
process encapsulation could be achieved.
Support for blocking instructions – If the traced process happened to block at a specific
instruction (e.g. an I/O instruction) the application used to freeze until the instruction was
finished. To avoid this situation we had to redesign the Process Tracer to make it use
asynchronous GDB commands for manipulating the traced process.
“Back” Feature – This feature is meant to memorize the states the process passed through
and give the opportunity to restore a previous state of the traced process.
Process Input – The GUI displays a container for each traced process that allows introducing
text that would be sent to process on its standard input.
Process Output – Also the GUI displays a container that shows the output of the traced
process.
Process Tree – This feature helps the user to visualize in a tree form the processes created
by the traced process. The user can select any of the created processes and see details
about its state.
Command line arguments – InspectOS is able to run the program with the given command
line arguments. The user is displayed a text field that allows him/her to introduce them.
Action Logging – At the button of the main window there appears a text area that displays
all user actions and the events.
Alexandru Șutîi
4 Implementation
4.1 Underlying Technology
The InspectOS modules are build using some libraries and tools. I will emphasize on the description
of the technology used by the user space modules, as our work in InspectOS was mainly oriented to
improvement of this parts.
4.1.1 GTK
All the Graphical User Interface was developed using a Python binding of GTK library called pyGTK.
This is a ubiquitous library on Linux. Although it contains the basic needed widgets, it lacks more
sophisticated widgets for drawing pages, graphs, trees, etc. Therefore some other graphical libraries
had to be added for displaying this kind of structures.
4.1.2 Graphviz and Cairo
For drawing the physical and virtual memory tables, Cairo was chosen. It has the necessary methods
for drawing this kind of structures. Graphviz is the tool used for drawing graphs. The input graph is
given in its specific description language and it provides an output image displaying the graph. That
image is taken by our GUI and displayed in a frame.
4.1.3 GDB
In order to manipulate the traced process we implemented our tracer over the known GNU
debugger GDB.
For every traced process we create a GDB instance [10]. If the process forks new processes they
remain in the same GDB instance. The Tracer module sends the GDB commands via the Expect
interface. Expect is implemented in TCL programming language for interacting with running
processes (in this case GDB). The tracer gets the GDB response via Expect, too, and takes the
appropriate action.
InspectOS – Extending Features for Operating System Inspection
4.2 Graphical User Interface Redesign
The OS-Tool developers chose a bottom up development mode, i.e. they first of all thought how to
gather and organize the information about the traced process from the OS, and only after that they
came with a way to display this information in the user interface. In consequence, the GUI was not
as intuitive for the beginners as it should be for an educational application. When we began working
on InspectOS our first goal was to redesign the user interface so that it was easier to understand and
encapsulate all the process data in the same graphical container.
In Figure 2 you can see the diagram of the OS-Tool GUI and Figure 3 displays a screenshot of the
graphical user interface. As you can see, the main window contained five main containers: the
source container, the VMA container, the memory container, the file descriptor table container and
the “Next” buttons container.
Figure 2 OS-Tool GUI Design
Alexandru Șutîi
Whenever a new process was spawned by the fork() system call, new frames were added to Source
and Memory containers and a new “Next” button was created. Also the VMA and File Table tabs
showed only the last process’s table.
Figure 3 OS-Tool Graphical User Interface
We did not find this interface well organized and we decided to have a better per process
organization. Therefore we reorganized the GUI as seen in the UML diagram from Figure 5 and in
the screenshot from Figure 4.
Figure 4 - InspectOS Screenshot
InspectOS – Extending Features for Operating System Inspection
The window contains two main frames. The frame in the right displays the physical memory. It
displays the physical memory frames for all the traced processes. In the right frame there is a pane
that allows the visualization of maximum two processes at the same time (you can select the
processes to view). All of the process resources are encapsulated in the same container.
The whole window is created in the Main class. When first opening the application, it displays a form
that prompts for the program to trace. For the each newly created process the Main object creates a
ProcessView object. This object encapsulates all the process information and returns a frame to be
displayed by Main. The Main object creates a VMA, a COW and a FileTableTab object and passes
these objects to all the new created ProcessView Objects. Also all the child processes created from a
traced process share the same Tracer object as they all of them reside in the same GDB session.
Whenever the ProcessView object has to display its frame, it gets its virtual memory table frame
from the COW object, its file descriptor table from the FileTableTab object and the VMA frame from
the VMA objects, by passing them its process id. Also it adds to the process frame the source, the
standard input and output frames.
Figure 5 - InspectOS New GUI Design
Alexandru Șutîi
4.3 GDB Asynchronous Commands
At the beginning the process tracer used only synchronous commands for interactions with GDB. It
worked well as long as there was no blocking instruction in the traced process, such as I/O
operations, synchronization object operations (semaphores, mutexes, waiting queues). When the
user was pressing the “Next” button, the tracer waited for the GDB response. If the process had
blocked in an instruction, the GUI thread also blocked, therefore the whole application was getting
frozen.
We had a lot of time searching the web for a solution to this problem. However, we did not find a
solution. So we wrote an email to the GDB users list to ask for help. This way we found out about
the GDB asynchronous commands.
So we have redesigned the tracer to send asynchronous commands and interrogate the status in
order to see if the current instruction ended. The detailed flow can be seen in the Figure 6. When
stepping the process the “step&” command is sent to GDB. The GDB prompt immediately returns as
this is an asynchronous command. In order to check the actual state of the process, the tracer sends
the “where” command. If the response contains the string “Target is executing” it means the
process is still blocked in the stepped instruction. To be sure that a non blocking instruction is not
perceived as blocking, the tracer waits 0.5 seconds before requesting the status. In this time the non
blocking instruction will end for sure.
Figure 6 - GDB asynchronous command flow
When the tracer announces that process is blocked, the GUI changes the “Next” button’s caption to
“Blocked” for a small amount of time. The next time the user presses “Next”, the tracer will check
again whether process is still blocked or not.
InspectOS – Extending Features for Operating System Inspection
In order to prepare GDB for asynchronous commands we have to input the following command
when starting the GDB instance:
> gdb ./program
[…]
(gdb) set target-async on
4.3.1 Another Possible Implementation
For the first time we thought to create a separate thread whenever a process got blocked. This
thread would have had to send “where” or “info program” commands in order to check whether the
process unblocked, and after that update the “Next” button’s state. We have renounced this idea as
adding multithreading to the application would have added a lot of new problems regarding thread
synchronization. Furthermore, this would have forced us to implement a safe access to the GDB
session. This would have been a bad idea, as we found a lot of bugs in GDB and this feature would
have revealed a lot more. Also it would have increased the used processing power.
4.3.2 A Known Issue
We have found a bug related to GDB asynchronous commands that we could not solve. The scenario
is as follows. Say we have a process A that blocks in reading from a pipe. The process B has to write
something in that pipe, causing the process A to unblock. However if we step the process B a few
times, but not enough that it would get to the instruction that will write in the pipe, and we step
again the process A (that is being blocked in the read instruction), the GDB sends a SIGKILL signal to
both the processes.
In order to solve this issue, we tried to get the process’s blocking state before sending a “step&” to it
again. But this approach was unsuccessful, as GDB takes all the debugged processes into the
“Stopped” state (stopped at a breakpoint), rather than leaving the blocked processes in “Sleeping”
state. Therefore, we could not find out the status of a process without resending the “step&”
command to it. The same problem occurs with multiple threads in the same situation.
4.4 Thread Support
An interesting and a challenging feature we came with, is the thread support. We first of all thought
that it would be useful for the users to visualize on real examples, problems like producerconsumer.
Threads in a process share the same address memory, the same VMA and the same file descriptor
table. The only private properties and resources of a thread are: the stack, the registers, current line
number (this is actually also stored in a register), the thread private data. Therefore we didn’t need
to change the memory, VMA and file table views as they are the same for all the threads in a
process.
Alexandru Șutîi
Figure 7 - Thread Diagram
We chose to display the line number for each thread and the variable values. Also InspectOS is able
to step each thread.
For every time a new thread is created, GDB announces this and the ProcessView object adds a new
entry for the newly created thread. The entry contains a thread id, a line number and a next button.
You can find a diagram in the Error! Reference source not found. and a screenshot of how the GUI
looks like. The thread TID is displayed in the left of its current line. Whenever the “Next” button is
pressed the tracer gets the TID of the thread, switches the thread number in GDB and sends the
“step&” command.
InspectOS – Extending Features for Operating System Inspection
Figure 8 - Thread Screenshot
As I have mentioned in the previous section, the thread support meets the same issue as for the
situation when there are more processes open. You have to step a thread until it unblocks another
before stepping the blocked thread again. Otherwise, GDB will kill all the threads.
We focused first of all on making the thread support feature work for the producer-consumer
problem. Therefore we have not tested it much for processes that both create processes and
threads.
4.4.1 Problems encountered
Whenever the process being debugged creates a new thread or process the GDB notifies by saying
“New process/thread created”. So to step a specific thread/process you have to switch to that
thread by entering the command “thread thread_no”. When you step a process, all the other
processes in the GDB session stay stopped.
However, GDB was not performing the same with the threads. Whenever you step a thread, the
other threads of that process begin running as well. So we had to figure out how to make them stay
stopped. In order to solve this problem we have to enter the following commands:
> gdb ./program
[…]
(gdb) run …
(gdb) set scheduler-locking on
[…]
Alexandru Șutîi
4.5 Process Standard Input
In its initial stage InspectOS was not designed to interact in any way with the traced process.
Therefore, we considered it necessary to provide a means to interact with the process’s input and
output and to pass the process with initial command line arguments.
In order to send input to the process, we have tuned the GUI and the Tracer modules. You can see a
diagram in the Error! Reference source not found.. In the GUI we added a frame that provides a text
box where the user can input the text and submit it.
Figure 9 - Input Flow
In order to make the process receive our input we had to redirect its standard input from a named
pipe. When the process is to be traced, InspectOS creates a named pipe. After this, it creates a GDB
session for the given process with the following commands:
> touch named_pipe
> gdb ./process
[…]
(gdb) run < named_pipe
When the user inserts some text in the input frame, the GUI sends the text to the Tracer by calling
its send_text() method with the text as an argument. The Tracer opens the write edge of the named
pipe and writes the text to it.
4.5.1 Known Issues
This approach is the only one we have thought of. Although, it works well for most of the cases, it
has some drawbacks.
If a process creates more processes with fork() system calls, all the spawned processes have the
same named pipe redirected to the standard input. Therefore, the user has to be careful that the
given text goes to the intended process. Moreover, if a process performs a read from the input, it
flushes all data from the pipe, even if it does not need the whole of it.
InspectOS – Extending Features for Operating System Inspection
4.6 Process Standard Output
4.6.1 The First Failed Approach – Redirect the Process Output
We have had a lot of headaches developing this feature. First time we thought to implement it the
easy, and we though the normal, way: by redirecting the output to a named pipe, and reading it
from the pipe, as we did with the standard input.
We realized that GDB has several ways of flushing the output for de debugging process. If you debug
the program without redirecting the output (its output is displayed in the terminal), GDB flushes
every line to the terminal every time the process outputs a new line. However, when redirecting the
output to a file, GDB realizes that this is not a terminal and flushes blocks of data, i.e. it buffers many
lines, and flushes them when the buffer is full. As a result we didn’t get the output from the pipe
immediately after it was written. So we renounced this solution.
4.6.2 The Second Failed Approach – Parse the GDB Output
The second approach was not very reliably, however it worked for most of the situations. This time
the Tracer was parsing the GDB output. We implemented a way to differentiate between the output
of the GDB commands and the output of the traced process. In most of the cases the GDB
commands’ output was following an easy patter so we could easily differentiate it. However things
got worse when we implemented the support for blocking instructions, i.e. using the GDB
asynchronous commands. The GDB output began to be less organized and we were failing to
differentiate the process output very well. Therefore, we thought to the last and the good solution
presented in the next section.
4.6.3 The Final Solution
We have found that there is a way to redirect the GDB output to a file, i.e. the output of the GDB
commands, while the output of the process being debugged is displayed in the terminal. This way
we do not have to parse the GDB output, we now know that we can get the process status (current
line, running status, etc.) from the redirected file, and in the terminal we get the pure output of the
process.
In order to enable this behavior we give the following GDB commands at the beginning of the debug
session:
> gdb
(gdb)
(gdb)
(gdb)
./program
set logging redirect on
set logging file redirect_file
set logging on
Whenever the “Next” button is pressed, the ProcessView object calls the step() method of the
Tracer by passing the process id. The Tracer sends the “step&” commands to the GDB. If the process
has not got blocked, the Tracer reads the response of the “step&” command from the redirection
file and returns the status of the process to the ProcessView (i.e. current line, blocked/not blocked,
Alexandru Șutîi
process created, thread created, process terminated, etc.). Also the tracer reads the output from the
terminal and saves it internally in a list of lines.
After the step() method ends, the ProcessView asks for the output from the Tracer and appends the
new output lines to the Output Frame text field.
Figure 10 - The Output flow
4.7 Variable Value Display
When we implemented the thread support, in a problem like Producer-Consumer the user could
only learn about the thread synchronization. However, this was not enough. So we decided to
implement the feature that would display the values of the variables, both global and local.
We just used the GDB “print” command that takes a variable name and returns its value. When the
user wants to find the value of a variable he is displayed a frame that contains a text input field
where he can give the variable name, a button and a read only text field where the value is
displayed. When the user presses the button, the ProcessView object calls the get_variable_value()
method of the Tracer with the variable name. The Tracer sends the print command to GDB and gets
the value that it returns to the ProcessView object.
The event flow can be better seen in Figure 11.
InspectOS – Extending Features for Operating System Inspection
Figure 11 - Print variable value command flow
Alexandru Șutîi
5 Test Scenarios
5.1 Compiling and Tracing a Process
In order to trace a process you have to have the source code and compile it with debugging symbols.
You can see an example of source code below:
#include <stdio.h>
#include <string.h>
#include <assert.h>
#define PAGE_SIZE (1024 * 4)
int
main(void)
{
char buf[5 * PAGE_SIZE];
memset(buf, 'x', PAGE_SIZE);
memset(buf + PAGE_SIZE, 'x', PAGE_SIZE);
memset(buf + 2 * PAGE_SIZE, 'x', PAGE_SIZE);
memset(buf + 3 * PAGE_SIZE, 'x', PAGE_SIZE);
memset(buf + 4 * PAGE_SIZE, 'x', PAGE_SIZE);
return 0;
}
You can compile this example by entering the following commands:
> gcc –g –Wall –o fault fault.c
The resulted binary can be selected in InspectOS and traced. While tracing the process, you are
displayed the source code and you can step the process. Note that the source file has to be in the
same folder with the binary so GDB could find it.
5.2 Unit Tests
In order to test InspectOS we have created a folder called examples in our repository [11]. For every
implemented feature we have a unit test that resides in this directory. This directory is a good start
for our users, in order to make them understand where to start from when using InspectOS. I will
present below the tests used for the basic features.
5.2.1 Memory Management
The following source code is a very good example to show the way the physical memory frames are
allocated while running the process.
1
2
3
4
5
6
7
#include <stdio.h>
#include <string.h>
#include <assert.h>
#define PAGE_SIZE (1024 * 4)
int
InspectOS – Extending Features for Operating System Inspection
8 main(void)
9 {
10
char buf[5 * PAGE_SIZE];
11
char *pbuf;
12
memset(buf, 'x', PAGE_SIZE);
13
memset(buf + PAGE_SIZE, 'x', PAGE_SIZE);
14
memset(buf + 2 * PAGE_SIZE, 'x', PAGE_SIZE);
15
memset(buf + 3 * PAGE_SIZE, 'x', PAGE_SIZE);
16
memset(buf + 4 * PAGE_SIZE, 'x', PAGE_SIZE);
17
pbuf = (((unsigned long)&buf[0]) & ~(PAGE_SIZE-1)) + PAGE_SIZE;
18
fork();
19
pbuf[PAGE_SIZE] = 'x';
20
pbuf[0] = 'x';
21
return 0;
22 }
At line 10 a buffer of five pages is allocated, that means only 5 pages of virtual memory are
allocated. The physical pages are allocated one by one from the line 12 to line 16. For each of these
lines a demand paging event is generated.
At the line 18 a new process is created. Five virtual pages are allocated for the child process and the
physical memory pages are marked read-only and shared between the two processes. At the lines
19 and 20 when the first process writes the second and the first page, two Copy-on-Write events
occur and two more physical pages are allocated for the child process.
5.2.2 File Descriptor Table
In order to test our File Descriptor feature we use the following program:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#include
#include
#include
#include
<stdio.h>
<fcntl.h>
<sys/stat.h>
<sys/types.h>
int
main(void)
{
int fd1, fd2, fd3, fd4, fd5;
fd1 = open("/tmp/a",
fd2 = dup(fd1);
close(0);
fd3 = open("/tmp/b",
close(1);
close(2);
fd5 = open("/tmp/a",
fd4 = open("/tmp/c",
O_RDONLY);
O_RDONLY); /* /tmp/b is a hard link to /tmp/a */
O_WRONLY);
O_WRONLY); /* /tmp/c is a soft link to /tmp/b */
return 0;
}
When the program starts, the descriptors 0 (standard input), 1 (standard output) and 2 (standard
error) are open. At line 11 the file descriptor 4 is being opened. At line 12 a new descriptor (5) points
to the file opened at line 11. Line 13 closes the standard input file descriptor. Line 14 opens file
descriptor 0 to point to /tmp/b. Lines 15 and 16 close the descriptors 1 and 2 (the standard output
and error). Line 17 opens file descriptor 1 to point to new file structure that point to /tmp/a. Line 18
Alexandru Șutîi
opens the file descriptor 2 to write to /tmp/c. This means the standard output and error will be
redirected to /tmp/a, respectively to/tmp/c. Standard input is redirected from /tmp/a.
5.2.3 Asynchronous Commands
In order to test that the GDB asynchronous commands work as expected and that the application
realizes when there is blocking instruction, we use the following unit test. The main process creates
a pipe. At the line 18 it creates a new process. The child process is being stepped till the line 23
where it should block as it wants to read from the pipe. Its “Next” button has to show Blocked until
the parent process gets to write in the pipe at the line 33. When the parent gets to the line 35 its
button has to show Blocked until the child process exits at the line 26.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#include
#include
#include
#include
#include
<unistd.h>
<stdio.h>
<stdlib.h>
<string.h>
<sys/wait.h>
int main()
{
int pipefd[2];
int pid, status;
int dummy = 0;
char hello[] = "Hello";
char recv[20];
if (pipe(pipefd))
perror("create pipe");
pid = fork();
if (pid == 0)
{
dummy = 2;
close(pipefd[1]);
read(pipefd[0], recv, 19);
printf("Got %s\n", recv);
close(pipefd[0]);
exit(0);
}
else if (pid < 0)
perror("fork");
dummy = 3;
close(pipefd[0]);
write(pipefd[1], hello, strlen(hello)+1);
close(pipefd[1]);
waitpid(pid, &status, 0);
return 0;
}
5.2.4 Thread Support
As I have explained earlier, the main purpose of implementing the thread support was to illustrate
the Producer-Consumer problem using threads. Therefore, we created a test example that creates
two threads, a producer and a consumer, that share a global buffer of five elements. The producer
InspectOS – Extending Features for Operating System Inspection
inserts randomly generated numbers and the consumer consumes them. This example is good for
testing the asynchronous commands and for displaying the variable values. In this problem the user
is interested to see when the threads stop at the synchronization points and the values from the
buffer.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
#include
#include
#include
#include
#include
<pthread.h>
<unistd.h>
<stdio.h>
<stdlib.h>
<string.h>
#define BUFFER_SIZE
#define NR_ITERATIONS
5
30
typedef struct {
int buff[BUFFER_SIZE];
int first;
int count;
} buffer_t;
void init_buffer(buffer_t* b) {
memset(b->buff, 0, sizeof(int) * BUFFER_SIZE);
b->count = 0;
b->first = 0;
}
void insert_item(buffer_t* b, int item) {
b->buff[(b->first + b->count)%BUFFER_SIZE] = item;
b->count++;
}
int remove_item(buffer_t* b) {
int item = b->buff[b->first];
b->first = (b->first + 1)%BUFFER_SIZE;
b->count--;
return item;
}
int is_buffer_full(buffer_t* b) {
return b->count == BUFFER_SIZE;
}
int is_buffer_empty(buffer_t* b) {
return b->count == 0;
}
/*
* the buffer where the producer will place items
* and from which the consumer will take items
*/
buffer_t common_area;
pthread_cond_t buffer_not_full = PTHREAD_COND_INITIALIZER;
pthread_cond_t buffer_not_empty = PTHREAD_COND_INITIALIZER;
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
void* producer_fn (void* arg) {
int item_to_insert, i;
for(i = 0; i < NR_ITERATIONS; i++) {
Alexandru Șutîi
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
item_to_insert = rand() % 100;
pthread_mutex_lock(&mutex);
// Wait the buffer not to be full
if (is_buffer_full(&common_area))
pthread_cond_wait(&buffer_not_full, &mutex);
// Insert the item
insert_item(&common_area, item_to_insert);
// Signal buffer not empty
if (common_area.count == 1)
pthread_cond_signal(&buffer_not_empty);
pthread_mutex_unlock(&mutex);
}
return NULL;
}
void* consumer_fn (void* arg) {
int item_consumed, i;
for(i = 0; i < NR_ITERATIONS; i++) {
pthread_mutex_lock(&mutex);
// Wait the buffer not to be empty
if (is_buffer_empty(&common_area))
pthread_cond_wait(&buffer_not_empty, &mutex);
// Consume the item
item_consumed = remove_item(&common_area);
// Signal buffer not full
if (common_area.count == BUFFER_SIZE-1)
pthread_cond_signal(&buffer_not_full);
pthread_mutex_unlock(&mutex);
}
return NULL;
}
int main ()
{
pthread_t producer_th, consumer_th;
int rc;
/* initialization */
init_buffer(&common_area);
srand(time(NULL));
/* create the threads */
rc = pthread_create(&producer_th, NULL, producer_fn, NULL);
if (rc != 0)
perror("pthread_create");
rc = pthread_create(&consumer_th, NULL, consumer_fn, NULL);
if (rc != 0)
perror("pthread_create");
/* wait for the threads to finish execution */
rc = pthread_join(producer_th, NULL);
if (rc != 0)
perror("pthread_join");
rc = pthread_join(consumer_th, NULL);
if (rc != 0)
perror("pthread_join");
return 0;
}
InspectOS – Extending Features for Operating System Inspection
5.2.5 Process Output
The unit test for the output feature is not a hard one. The traced process creates a child process.
Both of them print text to the output. The GUI has to know how to separate the output of each
process and show it in its output frame. Also the processes have to synchronize.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#include <stdio.h>
#include <sys/wait.h>
int main()
{
int pid;
printf("Started the process\n");
pid = fork();
if (pid == 0)
{
printf("The child process is created\n");
exit(0);
}
else if (pid < 0)
perror("fork");
printf("Child process created\n");
waitpid(pid, &status, 0);
printf("Child process exited\n");
return 0;
}
5.2.6 Process Input
In order to test the process input feature we use a test program that creates a child process and
both of them read integers from the input. This example shows the problem I have iterated in the
implementation section. If you enter more integers at a time, one process reads them all even if it
does not need all of them (it flushes the input pipe).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main() {
int read_value;
int nr_steps;
int i;
printf("Give the number of steps\n");
scanf("%d", &nr_steps);
if (fork() < 0)
perror("fork");
for (i = 0; i < nr_steps; i++) {
printf("Give a value to read.\n");
scanf("%d", &read_value);
printf("Read %d.\n", read_value);
}
printf("Exiting...\n");
return 0;
}
Alexandru Șutîi
6 User Feedback
6.1 InspectOS Developers’ Feedback
Before we started the development, we decided to put ourselves in the position of a project’s user,
i.e. the student that has pour operating system concepts knowledge. Without looking over the
source code we had to compile the project, run it and come with some objective impressions. This
was a productive thing to do, as we came with a list of bugs and issues that were to be discussed [2].
First of all we did not like the graphical user interface organization. It took some time to figure out
what every widget mean. So we decided to redesign the GUI as the first thing.
We also looked again over the operating systems labs and tried to identify new concepts that could
be visualized in InspectOS. This way we came with the ideas to implement the thread support, as it
is one of the hardest concepts to comprehend by the students. Also we decided to implement the
visualization of the process standard output and error and the created process tree. The idea to
implement a way to send text to the standard input came later.
When tracing processes in OS-Tool we often needed to see how the memory or the file table looked
at a previous moment. This led us to the idea to implement the “Back” option that would create
snapshots of the process resources and be able to display them on demand.
6.2 Feedback from the OS-Tool Developers and our
Supervisors
After finishing the proposed ideas list, we had a meeting with Andrei, Lucian and Vlad, the OS-Tool
developers, and presented them the list and the way we wanted to approach the implementation of
that features.
Although we had read their bachelor’s thesis and learned about the problems they have confronted
with while developing OS-Tool, this meeting helped us a lot. Lucian brought to our knowledge that
new Linux kernels provide the same information about a process without the need of the OS-Tool
kernel module. So we had been given the idea to renounce the kernel module. We did not do this,
as there were features more important to implement and to improve, such as the GUI redesign.
However, this could be accomplished in future development. Also Andrei and Vlad shared some
thoughts of how to redesign the GUI and what tools exactly to use.
A very objective feedback came from our supervisor Razvan Deaconescu, whom we presented our
work weekly. He came with a lot of great ideas regarding the thread support, GUI redesign, “Back”
feature.
InspectOS – Extending Features for Operating System Inspection
6.3 Feedback from Students
As InspectOS is an educational application, the most important feedback is the one from its future
users, the students. We have sent a presentation email to a list of students and asked them to try
InspectOS and respond with a valuable feedback.
The feedback was a really good one. We have received some improvement idea, and bug reports,
that we have solved. There were some small issues related to the GUI, but in great the users had a
good opinion about the application and found it helpful in understanding the operating systems
concepts. From this feedback we like to think that InspectOS is a stable and useful application at the
moment.
Alexandru Șutîi
7 Conclusions and Future Plans
From the beginning we planned to realize some new features and improvements. We are confident
that we have realized the most part of what we intended. The GUI redesign and the refactoring of
the GUI class hierarchy are one of our greatest realizations in this project. I believe that we have
created a better environment for our other features development and it will open new
opportunities for who is going to continue this project in the future.
The thread support feature seemed to me hard to realize, but after a lot of research and many tries I
feel confident that we have done a great job. The redesigning of the tracer to support asynchronous
GDB commands was also a great challenge that needed some research as well. Although the
implementation has some known issues, it works well for most of the cases.
Another feature we planned and realized is the “Back” feature. It suffered a lot of modifications until
it got to the current state, but the final result is a great one. I congratulate Radu for accomplishing
this goal of InspectOS.
Although we believe we accomplished most of our goals, there have remained some ideas that we
didn’t implement. Some of them are:





Create a full documentation for the GUI and add a Help menu entry.
Create a configuration file.
Create an installer (although we could implement this in the near future).
Although the file table shows the file types descriptors point to, we wanted to add more
details to this.
Improve the process stepping. This feature should allow placing break points and adding
more buttons (besides the Next button) that a normal debugger has.
These ideas can be found in our project’s issue list, too [2]. They can serve as a good starting point
for future development. Also the OS-Tool developers’ ideas that we haven’t accomplished can be
implemented by the future developers.
I feel very happy to have worked with my colleague Radu Velea. I want to thank him and our
supervisors for the interesting collaboration, as it was a successful one. Although we have
encountered several obstacles during our development, we have managed to surpass most of them.
I have learned a lot of new things and improved my knowledge and skills.
InspectOS – Extending Features for Operating System Inspection
A. Compiling, Installing and Running
InspectOS
System Requirements
In order to run InspectOS you need a Linux operating system with a 2.6 kernel. The kernel must
support Kprobes [15]. You also need python 2.6 to run the application.
Installation
Cloning the repository
To clone the InspectOS repository you have to have Git [16] installed on your computer. You can
create a read-only clone by entering the following command in the shell:
> git clone http://ixlabs.cs.pub.ro/git/inspectos.git
Installing auxiliary packages
There are a couple of third-party libraries and tools that have to be installed for InspectOS to run.
o
Python-rsvg [17]
o
> sudo apt-get install python-rsvg
RBTree module [18]
o
> wget http://newcenturycomputers.net/projects/download.cgi/RBTree-1.6.zip
> unzip RBTree-1.6.zip
> cd RBTree-1.6
> sudo python RBTree.py install
Graphviz [19]
o
> sudo apt-get install Graphviz
GDB [20]
> sudo apt-get install gdb
Running InspectOS
First of all you need to compile the kernel module and insert it. For this you need to enter the
following commands:
> cd $INSPECTOS_ROOT/fault
> make
> sudo insmod os-mon.ko
Alexandru Șutîi
Now you are ready to run the application. You simply need to go to InspectOS root directory and run
the script inspectOS.sh.
> cd $INSPECTOS_ROOT
> sudo bash inspectOS.sh
Tracing a process
When InspectOS starts you are displayed a form that lets you choose the program you want to
trace. You have to choose a binary that was compiled with debugging symbols and resides in the
same directory with its source code. You also can trace more processes by accessing Menu->File>Open.
When you are running InspectOS you are displayed a container for each process. Every process is
identified by its process id. You can find it in the Next button, in the container name, etc. You can
select what process to visualize by checking its checkbox in the up left corner of the window. You
can visualize maximum two processes at the same time.
In the right of the main window you can see the physical memory. Each process container contains
Next buttons for each thread, a source code frame, an input and an output frame, a virtual memory
frame, a VMA frame and a File Table frame. You can choose the frame to view by selecting the
appropriate radio button. There is also a Print button for printing the variables’ value.
InspectOS – Extending Features for Operating System Inspection
B. InspectOS Repository Description
I will briefly describe some of the directories that form our repository.





fault/ - This is the directory containing the source code for the kernel module. This is where
to compile the module and insert it.
examples/ - All our unit tests can be found here. This is a good starting point for the users
that use our application for the first time.
ixia-slides/ - This directory contains the presentation slides, logos, images.
styles/ - In order to give a custom color (yellow) to the current line in the source code frame
I had to extend a style. This style resides in this directory.
src/ - This is the source directory for the user space modules. The main directories and files
are:
o src/main.py – This file defines the Main class, the one that creates the window and
starts the application.
o src/ostool and src/os_tool/event – The processing module files reside in these
directories. Here are the classes that read the VMA, memory tables and file table
information from procfs and present it to the GUI.
o src/os_tool/gui – The GUI classes can be found in this directory. The file
process_view.py defines the ProcessView class that encapsulates all the information
about a process. It gets the VMA, memory and file descriptor frames from the
classes COW, VMA and FileTableTab that are defined in the files cow.py, vma.py and
file_table_tab.py respectively.
o src/os_tool/tracer.py - This file defines the Tracer module.
Alexandru Șutîi
References
[1] Wikipedia. Operating system - Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/Operating_system , 2011. [Online; accessed 1-July-2011].
[2] InspectOS issue list - https://ixlabs.cs.pub.ro/redmine/projects/inspectos/issues.
[3] Wikipedia. Process (Computing) - Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/Process_%28computing%29, 2011. [Online; accessed 1-July-2011].
[4] Wikipedia. Thread (Computer Science) - Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/Thread_%28computer_science%29, 2011. [Online; accessed 1-July2011].
[5] Wikipedia. Virtual Memory - Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/Virtual_memory , 2011. [Online; accessed 1-July-2011].
[6] Wikipedia. File descriptor - Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/File_descriptor , 2011. [Online; accessed 1-July-2011].
[7] Memory Management in Linux, http://www.makelinux.net/ldd3/chp-15-sect-1 , 2011. [Online;
accessed 1-July-2011].
[8 ] Wikipedia. Demand Paging - Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/Demand_paging , 2011. [Online; accessed 1-July-2011].
[9] Wikipedia. Copy-on-write - Wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Copyon-write , 2011. [Online; accessed 1-July-2011].
[10] R. Stallman, R. Pesch, S. Shebs, “Debugging with GDB”.
http://www.tutok.sk/fastgl/download/books/DebuggingWithGDB.pdf
[11] InspectOS Git Repository http://ixlabs.cs.pub.ro/git/inspectos.git
[12] A. Silberschatz, P.B. Galvin, G. Gagne, “Operating Systems Concepts – 7th Edition”.
[13] R. Love, “Linux Kernel Development”.
stid.googlecode.com/files/Linux.Kernel.Development.3rd.Edition.pdf
[14] GDB Mailing List, http://sourceware.org/ml/gdb/
[15] Kernel Probes Home Page, http://sourceware.org/systemtap/kprobes/
[16] Git Home Page, http://git-scm.com/
[17] Python rsvg Home Page, http://cairographics.org/pyrsvg/
InspectOS – Extending Features for Operating System Inspection
[18] RBTree Home Page, http://newcenturycomputers.net/projects/rbtree.html
[19] Graphviz Home Page, http://www.graphviz.org/
[20] GDB Home Page, http://www.gnu.org/software/gdb/