REMOTE DESKTOP PROTOCOL INTEGRATION
Lakshmi Nidadavolu
B.E., Nagpur University, 2005
PROJECT
Submitted in partial satisfaction of
the requirements for the degree of
MASTER OF SCIENCE
in
COMPUTER SCIENCE
at
CALIFORNIA STATE UNIVERSITY, SACRAMENTO
FALL
2011
© 2011
Lakshmi Nidadavolu
ALL RIGHTS RESERVED
ii
REMOTE DESKTOP PROTOCOL INTEGRATION
A Project
by
Lakshmi Nidadavolu
Approved by:
__________________________________, Committee Chair
Du Zhang, Ph.D.
__________________________________, Second Reader
Ahmed Salem, Ph.D.
____________________________
Date
iii
Student: Lakshmi Nidadavolu
I certify that this student has met the requirements for format contained in the University format
manual, and that this project is suitable for shelving in the Library and credit is to be awarded for
the project.
__________________________, Graduate Coordinator
Nikrouz Faroughi, Ph.D.
Department of Computer Science
iv
___________________
Date
Abstract
of
REMOTE DESKTOP PROTOCOL INTEGRATION
by
Lakshmi Nidadavolu
Currently few Remote Desktop Protocol (RDP) Clients are capable of presenting the
user’s windows desktop and giving the keyboard control over the client computer. The
existing RDP clients are interactive in nature. Each keystroke is sent to the remote
desktop in an interactive basis. Recently, there is more demand for automated tasks to
perform a set of actions on the remote desktop.
The aim of this project is to implement Remote Desktop Protocol Integration to perform a
set of actions on the remote desktop as an automated task with minimum manual
assistance. The integration includes the following main components.

RDP client takes file input
The client reads the file, then converts each character into a set of key
strokes as key pressed, key typed and key released and sends the
scancodes of the keys to the remote desktop in the same order.

Record user actions
The client records all the user actions along with the delays in between the
actions to a file. Actions recorded to the file are of the format:
v
(scan codes, key pressed / key released, time delay between consecutive
actions). The user selects a portion of the snapshot and compare to any
other saved snapshot/image based on the co-ordinates.

Playback actions
The client plays back the recorded actions from the saved file.
The existing RDP clients are not capable of performing a set of actions without manual
assistance on the remote server. Hence, RDP integration is an enhancement to an existing
RDP client called properJavaRDP. This integration is capable of performing a set of
actions on the remote server.
, Committee Chair
Du Zhang, Ph.D.
______________________
Date
vi
DEDICATION
To my family
vii
ACKNOWLEDGMENTS
I would like to thank my project advisor Dr. Du Zhang for taking time and giving me
guidance through the project.
I would also like to thank all the professors who taught me during my masters’ course
work.
I would like to thank my family for all the support.
viii
TABLE OF CONTENTS
Page
Dedication………………………………………………………………………………..vii
Acknowledgements .......................................................................................................... viii
List of Figures ................................................................................................................... xii
List of Tables ................................................................................................................... xiii
Chapter
1. INTRODUCTION ...................................................................................................... …1
1.1 Current state of practice for properJavaRDP client……………. …..………………1
1.2 Scope of Remote Desktop Protocol Integration.......................... …..………………1
1.3 Benefits of RDP Integration.................................................................................. …3
2. BACKGROUND ........................................................................................................ …4
2.1 Current exixting RDP clients ................................................................................ …4
2.1.1 rdesktop RDP Client ...................................................................................... …4
2.1.2 Seamless Java RDP Client ................................................................................. 5
2.1.3 Elusiva RDP Client ............................................................................................ 5
2.2 properJavaRDP Client .............................................................................................. 6
2.2.1 Keyboard shortcuts ............................................................................................ 6
2.2.2 Key map ............................................................................................................. 7
2.2.3 Process orders .................................................................................................... 8
3. ARCHITECTURE AND TECHNOLOGY .................................................................. 11
ix
3.1 Basic Architecture of RDP Integration ................................................................... 11
3.1.1 Communication steps ....................................................................................... 12
3.2 Features and Capabilities………………………………………………………….13
3.3 Technology used in RDP Integration ...................................................................... 15
3.3.1 Client-Server Connection................................................................................. 16
4. IMPLEMENTATION DETAILS ................................................................................. 19
4.1 File input ................................................................................................................. 21
4.2 Record actions ......................................................................................................... 22
4.2.1 Capture snapshots ............................................................................................ 23
4.2.2 Crop, confirm and save snapshot. .................................................................... 24
4.3 Playback actions...................................................................................................... 25
5. PERFORMANCE EVALUATION .............................................................................. 27
6. CONCLUSION AND FUTURE WORK ..................................................................... 29
6.1 Advantages .............................................................................................................. 29
6.1.1 Efficient and accurate ...................................................................................... 29
6.1.2 Visual confirmation ......................................................................................... 29
6.2 Future work ............................................................................................................. 29
6.2.1 Alerts on remote server .................................................................................... 29
6.2.2 Security ............................................................................................................ 29
6.2.3 Revert to initiation point .................................................................................. 30
6.2.4 Run parallel automated tasks ........................................................................... 30
Appendix Source Code .................................................................................................... 31
x
Bibliography ..................................................................................................................... 49
xi
LIST OF FIGURES
Page
Figure 1 Key map I ............................................................................................................. 7
Figure 2 Key map II ............................................................................................................ 8
Figure 3 Process rectangle orders ....................................................................................... 9
Figure 4 Process text orders .............................................................................................. 10
Figure 5 Architecture diagram .......................................................................................... 12
Figure 6 Wrapped image ................................................................................................... 15
Figure 7 TCP socket flow diagram ................................................................................... 17
Figure 8 Flow diagram of RDP Integration ...................................................................... 19
Figure 9 User options ........................................................................................................ 20
Figure 10 User file input ................................................................................................... 20
Figure 11 User authentication ........................................................................................... 21
Figure 12 Input file ........................................................................................................... 22
Figure 13 File to record actions ........................................................................................ 23
Figure 14 Capture snapshot .............................................................................................. 24
Figure 15 Crop snapshot ................................................................................................... 25
Figure 16 Performance evaluation chart ........................................................................... 27
xii
LIST OF TABLES
Page
Table 1 Keyboard shortcuts ................................................................................................ 7
xiii
1
Chapter 1
INTRODUCTION
Remote Desktop Protocol (RDP) [4] provides remote control on the windows computers
including both desktops and servers. RDP includes properties like encryption,
authentication, bandwidth, resource sharing over secure network communications
protocol. While running the RDP client, all the client-side execution, data processing,
data storage and data execution occurs on the server. All the keyboard inputs transmit to
the remote sever over the network from the client.
1.1 Current state of practice for properJavaRDP client
The properJavaRDP client initializes the RDP5 communication layer and then establishes
a connection with the remote server based on the username and password on the default
port 3389. After the license negotiation with the remote server, the client is notified that
the connection is ready to send messages to the remote server. It then pops up a window
frame showing the remote desktop on the client system. At this point, the properJavaRDP
identifies each keystroke interactively and then sends the scan code of the key to the
remote server. In addition, it logs the details showing whether the key is pressed, typed or
released for each keystroke.
1.2 Scope of Remote Desktop Protocol Integration
RDP integration is an enhancement to the properJavaRDP [2]. Instead of listening to each
keystroke, this integration translates a set of actions into respective keystrokes and sends
their scan codes to the remote server in the required order. Such functionality can be used
2
to automate tasks on the remote desktop. The automation of tasks on the remote server
can be achieved with the following capabilities of the RDP integration.
RDP integration takes a file as an input.

It reads the file one character at a time.

Determines the scancode of the character from the data file.

Sends the scancode to the remote server.

Also sends the value of whether the key is pressed, typed or released. These
values are sent in the required order.
RDP integration records user actions

The user can record the set of actions to a file that can be later automated on the
remote server.

Format of the actions recorded to the file is (scan codes, key pressed / key
released, time delay between consecutive actions).

The scan codes of the keys are stored in a file using the key listener.

The state of the key i.e. pressed, typed or released is stored next to the scan code
in the file.

The time delay between two consecutive scan codes is stored along with the later
scan code in the file.
RDP integration plays back the recorded actions.

Sends the scan codes to the remote desktop in the same order as in the file.
3

The time delay helps the actions to play back in the same time frame as they were
recorded to the file.
RDP integration captures snapshot of the remote desktop.

At anytime of recording the actions, the user can also capture the snapshot of the
remote server and view it on the client side.

The user can select a portion of the snapshot and compare it to any saved image.

The captured snapshot of the remote server is a confirmation to the user during
the play back of the recorded actions.
1.3 Benefits of RDP Integration
The remote desktops are commonly used for development and testing. Different options
are available to work interactively with the remote desktops. However, there is no option
to automate the work to be done on the remote desktop. Hence, the RDP integration
provides the feature to automate the frequent tasks on the remote desktop that can save
lot of time, effort and can be reused.
Chapter 2 includes the background on the Java RDP Clients. Detailed information on the
properJavaRDP client includes keyboard shortcuts, key mapping and processing orders
on the client. Chapter 3 includes the architecture and features of the Java RDP client
(properJavaRDP client). Also, it includes the technology used and the steps involved in
client-server connection. Chapter 4 includes implementation details of different modules
of the RDP integration. Chapter 5 includes the performance evaluation of the RDP
integration. Chapter 6 includes the conclusion and future work.
4
Chapter 2
BACKGROUND
There is extensive use of remote desktop protocol for most of the development, testing
and business purposes on the remote server [1, 4, 5].
This project is an enhancement to the properJavaRDP client. Most of the existing Java
RDP Clients are based on the rdesktop RDP client.
2.1 Current existing RDP clients
Some of the existing Java RDP clients are listed below [2, 9-11].
2.1.1 rdesktop RDP Client
rdesktop is an open source Java RDP client on an Unix platform. The latest version of
rdesktop includes bitmap caching, encryption, file system redirection, keyboard mapping,
audio redirection, login authentication, etc [10]. It helps to connect from an Ubuntu
machine to a windows machine. The windows machine can be remotely controlled from a
linux machine. The rdesktop is compared to the VNC in the following aspects.

rdesktop is faster when compared to the VNC in terms of performance.

The connection using VNC is unencrypted whereas; the rdesktop sends the login
details encrypted as network packets.

VNC can work on most of the operating systems.

VNC allows multiple sessions.
5
2.1.2 Seamless Java RDP Client
Seamless RDP client is based on the properJavaRDP client but in a seamless fashion. As
rdesktop RDP client, the seamless RDP client is also a RDP client on linux. Using
seamless RDP, the applications run locally on the client machine. The seamless client
allows connection sharing, i.e. it can launch multiple applications using a single
connection. The client allows rdesktop to run as a slave and notifies the master rdesktop
to run a new command. The applications launched by the seamless Java RDP client run
as local applications. Once the connection is established with the remote server, multiple
applications can be launched as local applications. It enhanced the properJavaRDP client
from AWT to Swing [9].
2.1.3 Elusiva RDP Client
Elusiva client is also an open source based on both rdesktop and properJavaRDP client. It
runs on windows, linux and other java enabled systems. It enables sound redirection and
allows virtualization [11].
Elusiva RDP client has the following features.

It is compatible with RDP 4.0, 5.0, 5.1, 5.2, 6.0, 6.1 and 7

It also supports clipboard mapping.

Enables encryption and security during communication.

Allows serial and parallel ports redirection.

Enables local audio.

Supports various keyboard shortcuts.
6
2.2 properJavaRDP Client
properJavaRDP is an open source Java RDP client based on rdesktop. The Remote
Desktop Protocol Integration is an enhancement based on the properJavaRDP client. The
properJavaRDP supports only clipboard transfer, but does not support file transfer. In
addition, it supports only single session on the remote server [3].
The debug options used in properJavaRDP are as follows.

--debug-key: At execution shows, the scan codes sent for each key press.

--debug-hex: At execution shows, the raw bytes sent and received over the
network.
2.2.1 Keyboard shortcuts
As shown in the table 1 [2] different key combinations are used on the remote server to
perform an action that conflicts with the client key combinations. For example, the tab
key would cycle through the recent windows on the client. The key combination to
enforce tab on the remote server is Alt+Insert. In this case, when the key listener detects
the key combination of Alt+Insert, then the client sends the scancode of tab key to the
remote server.
7
Table 1 Keyboard shortcuts
Ctrl+Alt+End
To open NT security dialog box.
Alt+Page Up
To switch between windows from left to right.
Alt+Page Down To switch between windows from right to left.
Alt+Insert
To tab through the recent windows.
Alt+Home
To open the start menu.
Alt+Delete
To open the context menu.
Ctrl+Alt+Minus To take a snapshot of the window that is active.
Ctrl+Alt+Plus
To take a snapshot of the entire screen.
2.2.2 Key map
The figure 1 shows the key map used to retrieve the information about the scan codes of
the keys entered. Key map includes key codes, scan codes and the name of the file.
Figure 1 Key map I
8
The figure 2 shows that the special keys have id 0 and all other keys have id 1.
Figure 2 Key map II
2.2.3 Process orders
The remote server sends the network packets containing orders to the client. A network
packet received by the client has n orders to process. The client interprets the orders as
primary and secondary. For example, the client receives the rectangle order as below.
X 11Y 58CX 58CY 15Color 15220
It consists of the x-y co-ordinates of the rectangle. It also includes the information about
the color to fill in the rectangle. After the client encounters the order for a rectangle, it is
processed by drawing rectangle and filling the rectangle with the given color.
For example, Alt+Home key is pressed to open the start menu. As shown in the figure 3,
the client draws the rectangle for the start menu.
9
Figure 3 Process rectangle orders
As shown in the figure 4, the rectangle is filled with the color received in the order. Next
steps include drawing the text on the rectangle of the start menu. In this way, each order
is processed on the remote server. To display the processed orders on the server, it uses
its own video card.
10
Figure 4 Process text orders
11
Chapter 3
ARCHITECTURE AND TECHNOLOGY
3.1 Basic Architecture of RDP Integration
The architecture of the RDP integration is same as the architecture of the properJavaRDP
client. Figure 5 [8] shows the client and the remote server components. After the
connection is established between the client and the server and the connection is ready to
send messages to the server, a window frame of the remote server pops up on the client.
The pop up window frame is represented as the “Server-side Application GUI” in the
Client component of the architecture diagram. The “Server-side Application” in the
Remote server component represents any application running on the remote server. The
“Server-side Application” can be viewed, controlled and manipulated using the “Serverside Application GUI” on the client.
The communication channel between the client and the server is on the default port 3389
and using remote desktop protocol. The remote server should be configured to allow
remote connections [6].
12
Figure 5 Architecture diagram
3.1.1 Communication steps
Figure 5 [7, 8] shows the communication between client and the remote server is over
default port 3389 and using remote desktop protocol. Following are the steps of
communication between the RDP client and the remote server.

The client sends keystrokes data over the network to the remote desktop based on
the key pressed, key typed and key released.

The remote server uses its own video driver to render display output.

The remote server constructs the information into network packets using RDP
protocol.

The remote server sends the network packets containing orders to the client. A
single network packet includes n orders.

The client interprets and processes these orders.
13

The client draws the required shapes based on these orders on the remote server.
The orders also include the color to fill in these shapes.
3.2 Features and Capabilities
The main features and capabilities of the RDP integration are listed below. These features
are based on the properJavaRDP client.

Key listener
RDP Client adds key listener to receive key events from the keyboard. The key
listener detects the action performed on a key using keyboard.

Encryption
RDP Client has an option to disable encryption from client to server. If encryption
is disabled, then only the packet with login information is encrypted and all the
other packets are delivered without encryption over the virtual channel. Both 40or 128-bit keys are supported for encryption [4].

Bandwidth
RDP Client has an option to enable/disable latency option for bandwidth
reductions by saving the tcp packets transmitted over a network connection.

Connection with remote server
A user can connect to the remote server using the RDP Client using the server
login credentials and by setting different logging levels like debug, warn, error,
etc. The user can disconnect the remote server without logging off.
14

Clipboard mapping
Users can transfer the clipboard data between applications running on the client
and those running on the remote session [4].

Virtual channels
RDP Client has an option specify the virtual channel to initialize the RDP
communication layer. The data packets process over the virtual channels.

Remote control
RDP Client shows the graphical interface of the remote server and enables control
over the remote server. The user can also specify the dimensions of the remote
window to view. It does not allow multiple sessions of the remote server [4].

Wrapped Images
The RDP Client receives a buffered image of the remote desktop in every few
seconds as shown in figure 6.
15
Figure
6 Wrapped
image
Figure
6 Wrapped
image
3.3 Technology used in RDP Integration
As the properJavaRDP client is based on java, the RDP integration is also developed in
java using the eclipse IDE. properJavaRDP can run on Java 1.1 and up.
The eclipse IDE is an open source integrated development environment. The downloaded
jar of the properJavaRDP 1.1 is included in the build path of a new java project in
eclipse. The RDP integration can be compiled and executed directly in the eclipse IDE as
a java application.
Java (TM) SE Development Kit 6 Update 22 is used for the development of RDP
integration. It is developed by Oracle and the version 1.6 of this product is used.
The Java (TM) SE Development Kit Update 22 includes the following.
16

The Java Runtime Environment (JRE)

Capable of developing applications and applets with the help of command line
tools like debuggers and compilers.
The RDP integration is tested against a virtual machine with a Windows Server 2008 R2
operating system. The virtual machine is created using the VMWare vSphere Client 4.1.
The windows test machine should have the feature to allow connections from computers
enabled. Also it should be verified that firewall settings of the test machine allows the
remote desktop connections to this system. The RDP integration can successfully execute
on any windows machine that has the remote connections enabled.
3.3.1 Client-Server Connection
The network socket is used for bidirectional communication. As shown in the figure 7
[12], the client and the server use the TCP socket communication to send and receive data
as network packets.
17
Figure 7 TCP socket flow diagram
The steps involved are as follows:

The TCP transport protocol is used for communication between client and server.

Initially the client sends a TCP connection request to the remote server with the
details of the client protocol version.

After establishing the connection, the client sends data to the server and the server
receives the data.

The server converts the received data into network packets and sends back to the
client.

The client then receives these network packets and then processes the orders
included in the packets.
18

After processing all the data, the client sends an end message to the server
requesting to close the socket.

The server receives the close socket message and finally closes the socket and
causes the RDP client to release the resources allocated to the client and server
socket.

Finally disconnects from a RDP session closing all the sockets.
19
Chapter 4
IMPLEMENTATION DETAILS
Figure 8 shows the overall flow diagram of the RDP integration. For all the features of
RDP integration, it transforms the input as scan codes that can be executed at the remote
server. The RDP integration finally uses the RDP client for sending the scan codes to the
remote server over network communications.
Figure 8 Flow diagram of RDP Integration
The RDP integration uses the methodology of FIFO (first in, first out). During
automation, the scan codes of the actions are sent to the remote server in the same order
as they were entered to the file. As shown below, buffered writer is used to write the scan
codes to a file specified by this.myString.
BufferedWriter writer;
20
writer = new BufferedWriter(new FileWriter(this.myString));
The RDP integration prompts the user to choose from the below listed options as shown
in figure 9.
1. Read from file
2. Record
3. Playback
Figure 9 User options
To read from a file, the input file is passed as a part of the arguments to the RDP client.
For the options record and playback, the user is prompted to enter the local file path as
shown in figure 10.
Figure 10 User file input
After the connection is established with the remote server, the user is prompted to enter
the credentials as shown in figure 11.
21
Figure 11 User authentication
4.1 File input
When a file as shown in figure 12 is given as an input to the RDP integration, the RDP
client reads each line from the file and determines the scan codes based on the key
mapping.
The file performs an automated task on the remote server that includes the following
steps.
1. Opens a notepad from the start menu.
2. Types the string “Hello World” in the notepad.
22
3. Saves the file from the file menu of the notepad.
4. Names the files as test1.
5. Closes the notepad window.
6. Deletes the test1 file.
7. Opens the command prompt from the start menu.
8. Types the dir command from the current location.
9. Closes the command prompt.
Figure 12 Input file
4.2 Record actions
Actions recorded to a file are of the format (scan codes, key is pressed / released, time
delay between consecutive actions) as shown in the figure 13. 0 indicates the key pressed
and 49152 indicates the key released. The file shown in figure 13 is the result of the
record actions feature of the RDP integration. This file would be the input to the playback
actions feature of the RDP integration.
23
While recording the actions to a file, the user can capture the snapshot at different
intervals. At the same time intervals, the snapshot will be displayed at the time of
automation.
Figure 13 File to record actions
4.2.1 Capture snapshots
At anytime of recording actions the key F8 is used to capture the snapshot of the remote
server.
if(scancode == 0x42 && diag == null){
diag = new wizard("Screenshot", bi_local);
The scan code of F8 key is 0x42. Only one JFrame window can be opened while
recording a set of actions. Hence in the above code, the condition if the JFrame object is
null or not. bi_local, a BufferedImage object is obtained from the wrapped images that
are sent to RDP client in every few seconds.
As shown in the figure 14, the menu item capture has an option to update the snapshot on
the JFrame window. Also using the mouse click the portion of the image can be selected
for comparison with any other image.
24
Figure 14 Capture snapshot
4.2.2 Crop, confirm and save snapshot
The capture menu also has an option to crop the snapshot. The figure 15 is the snapshot
after cropping the selected portion of the image in figure 14.
To compare and confirm based on the co-ordinates the below code is used. Here, ‘r’ is the
selected rectangular portion.
PixelGrabber g1 = new PixelGrabber(image1, r.x, r.y, r.width, r.height, false);
PixelGrabber g2 = new PixelGrabber(image2, r.x, r.y, r.width, r.height, false);
Then the pixels are retrieved based on the width and height of the rectangle as an integer
array.
25
if (g1.grabPixels()){
data1 = (int[]) g1.getPixels();
if (g2.grabPixels()){
data2 = (int[]) g2.getPixels();
Then both the images are compared pixel by pixel.
Figure 15 Crop snapshot
The file menu of the JFrame window also has the option to save the snapshot to the local
system.
4.3 Playback actions
The file shown in figure 13 is an input to the playback actions’ feature of the RDP
integration. The input file is scanned line by line. The three values in a line (scan code,
key pressed/ key released, time delay) are converted into tokens. Each respective token
value is retrieved as shown in the code below.
26
int time = Integer.parseInt(tokens[0]);
int act = Integer.parseInt(tokens[1]);
int code = Integer.parseInt(tokens[2]);
The above retrieved values are used to playback the recorded actions as automated task
on the remote desktop. During playback the snapshots of the remote desktop are
displayed at the same intervals as they were captured during the recording of the actions.
This gives a visual confirmation at different intervals of the execution of the automated
task.
27
Chapter 5
PERFORMANCE EVALUATION
The main advantage of the RDP integration is with respect to time, effort and usability.
RDP integration saves ample amount of time because it does not need manual effort to
perform tasks on the remote desktop. The tasks that are to be performed repeatedly on the
remote server can be recorded once using RDP integration. The recorded file can be
reused any number of times to perform same task frequently.
Figure 16 shows the comparison between the properJavaRDP client and the RDP
integration to perform the same set of actions against the same test machine (remote
server).
Figure 16 Performance evaluation chart
28
It represents the time taken to perform a set of actions using the RDP client on the remote
server using red. It represents the time taken to perform a set of actions using the
enhanced RDP integration in blue. The time of execution is represented in milliseconds.
As the figure 16 shows, there is not much time difference between the two. With respect
to the performance of the RDP integration, it takes about the same time.
The main difference between the original RDP client and the RDP integration is with
respect to the effort. The RDP client is only interactive in nature whereas, RDP
integration in automated in nature. Hence, there is less scope for errors and incorrect
results if the environment is set correctly. The only issue with the RDP integration is that
the user has no control over the automated task once it is initiated. The RDP client has
more control on the remote actions as it is more interactive in nature.
29
Chapter 6
CONCLUSION AND FUTURE WORK
The Remote Desktop Protocol Integration successfully accomplishes the goal to automate
tasks using file input, recording actions and playing back recorded actions.
6.1 Advantages
6.1.1 Efficient and accurate
The RDP integration efficiently runs the automated tasks on the remote server within the
estimated time. It saves time and effort with minimum manual interference.
6.1.2 Visual confirmation
Provides a snapshot of the remote server including its current state. The snapshot is an
accurate form of visual confirmation.
6.2 Future work
6.2.1 Alerts on remote server
There has to some kind of notification or alerts that gives user some information about
the other applications that initiated before the automated task started by the RDP
integration was completed.
6.2.2 Security
The RDP integration does not guarantee the security of the automated task running on the
remote server. There is a possibility of a windows application to interfere with the
running automated task.
30
6.2.3 Revert to initiation point
If the automated task has to end abruptly due to interference, then the user should have an
option to revert to the initiation point to avoid unwanted results.
6.2.4 Run parallel automated tasks
Currently the RDP integration is capable of running only one automated task at a time. It
should the capability to initiate multiple automated tasks to improve performance. It
should also be possible using command line option.
31
APPENDIX
Source Code
Input_Local.java
package com.hp.oo.content;
import java.io.*;
import java.util.Iterator;
import java.util.Scanner;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import net.propero.rdp.Input;
import net.propero.rdp.RdesktopCanvas;
import net.propero.rdp.Rdp;
import net.propero.rdp.keymapping.KeyCode;
import net.propero.rdp.keymapping.KeyCode_FileBased;
import net.propero.rdp.keymapping.MapDef;
/**
* @author Lakshmi
* Process all the keys and convert it to keystroke to determine its scan code.
*/
public class Input_Local extends Input{
StringBuilder contents = new StringBuilder();
String flag = null;
public Input_Local(RdesktopCanvas c, Rdp r, KeyCode_FileBased k){
super(c, r, k);
}
// New thread to scan each character from a file
class MyThread extends Thread{
public MyThread(){
}
public void run(){
File file = new File("C:\\data.txt");
try {
Scanner scanner = new Scanner(file);
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
32
line = line.replace("<#>", "\r");
line = line.replace("<##>", "\n");
try {
MyThread.sleep(2000);
inputStr(line);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
scanner.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
}
// This sends a signal that the connection is ready to send messages. Also //checks if
only one thread is running to process the file
public void triggerReadyToSend() {
try{
Thread mythread = new MyThread();
if(flag != "done")
{
flag = "done";
mythread.start();
}
}
catch(Exception e)
{
logger.debug(e.getMessage());
}
}
//Process all the special keys
public void inputStr(String str) {
boolean alt_pressed = false;
boolean ctl_pressed = false;
modifiersValid = true;
char[] ac = {'?','<','>',':','"','+','_','{','}','|','!','@','#','$','%','^','&','*','(',')'};
String[] strArr = str.split("[<>]+");
Pattern pattern = Pattern.compile("F[0-9]*");
int action = 1;
long time = getTime();
for(int l=0; l<strArr.length; l++){
33
try {
MyThread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
Matcher matcher = pattern.matcher(strArr[l]);
char[] arr;
if(strArr[l].length() > 3 && strArr[l].substring(0,
4).equalsIgnoreCase("alt-")){
sendScancode(time,0,56);
scancodeToFile(0,56);
alt_pressed = true;
arr = altCombinationKeys(strArr[l]);
if(arr == null)
continue;
}
else if(strArr[l].length() > 3 && strArr[l].substring(0,
4).equalsIgnoreCase("ctl-")){
sendScancode(time,0,29);
scancodeToFile(0,29);
ctl_pressed = true;
if(strArr[l].length() > 12 && strArr[l].substring(4,
8).equalsIgnoreCase("alt-") && strArr[l].substring(8, 14).equalsIgnoreCase("delete")){
sendScancode(time,0,56);
scancodeToFile(0,56);
alt_pressed = true;
sendScancode(time,0,211);
scancodeToFile(0,211);
continue;
}
else
arr = strArr[l].substring(4, 5).toCharArray();
}
else if(strArr[l].length() > 3 && strArr[l].equalsIgnoreCase("delete")){
sendScancode(time,0,211);
scancodeToFile(0,211);
continue;
}
else if(strArr[l].length() > 10 && strArr[l].substring(0,
6).equalsIgnoreCase("shift-")){
sendScancode(time,0,42);
scancodeToFile(0,42);
if(strArr[l].substring(6, 12).equalsIgnoreCase("delete"))
sendScancode(time,0,211);
scancodeToFile(0,211);
34
sendScancode(time,49152,42);
scancodeToFile(49152,42);
continue;
}
else if(strArr[l].length() > 0 && matcher.find()){
functionKeys(strArr[l]);
continue;
}
else if(strArr[l].equals("right")){
sendScancode(time,0,205);
scancodeToFile(0,205);
continue;
}
else if(strArr[l].equals("left")){
sendScancode(time,0,203);
scancodeToFile(0,203);
continue;
}
else if(strArr[l].equals("up")){
sendScancode(time,0,200);
scancodeToFile(0,200);
continue;
}
else if(strArr[l].equals("down")){
sendScancode(time,0,208);
scancodeToFile(0,208);
continue;
}
else
arr = strArr[l].toCharArray();
for(int i=0;i<arr.length; i++){
boolean setFlag = false;
for(int n=0; n<ac.length; n++){
if(ac[n] == arr[i])
setFlag = true;
}
int flags = 0;
action = 1;
String debugString = "Sending keypresses: ";
time = getTime();
int keycode;
int scancode=0;
char keyChar;
boolean upper = Character.isUpperCase(arr[i]);
if(upper || setFlag) {
35
debugString += "(0x"
+ Integer.toHexString(42)
+ ", "
+ ((action == KeyCode_FileBased.UP ||
action == KeyCode_FileBased.QUIETUP) ? "up": "down")
+ ((flags & KBD_FLAG_QUIET) != 0 ? "
quiet" : "")
+ " at " + time + ")";
sendScancode(time,0,42);
scancodeToFile(0,42);
logger.debug(debugString);
}
Iterator j = this.canvas.fbKeys.keyMap.iterator();
MapDef best = null;
MapDef current = null;
while (j.hasNext()) {
current = (MapDef) j.next();
if (current.appliesTo(arr[i])) {
best = current;
}
else if(Character.isWhitespace(arr[i])){
if(arr[i] == '\t') {
if(current.getKeyCode() == 9)
best = current;
}
if(arr[i] == '\n'){
if(current.getKeyCode() == 10)
best = current;
}
if(arr[i] == ' '){
if(current.getKeyCode() == 32)
best = current;
}
if(arr[i] == '\b'){
if(current.getKeyCode() == 8)
best = current;
}
if(arr[i] == '\r'){
if(current.getKeyCode() == 10)
best = current;
}
}
}
scancode = best.getScancode();
36
keycode = current.getKeyCode();
keyChar = best.getKeyChar();
logger.debug("PRESSED keychar='" + arr[i] + " char='"
+ keyChar + "'");
logger.debug("TYPED keychar='" + arr[i] + " char='"
+ keyChar + "'");
debugString += "(0x"
+ Integer.toHexString(scancode)
+ ", "
+ ((action == KeyCode_FileBased.UP || action ==
KeyCode_FileBased.QUIETUP) ? "up"
: "down")
+ ((flags & KBD_FLAG_QUIET) != 0 ? " quiet" : "")
+ " at " + time + ")";
sendScancode(time,0,scancode);
scancodeToFile(0,scancode);
logger.debug(debugString);
//To release the alt key if its pressed
if(alt_pressed){
sendScancode(time,49152,56);
scancodeToFile(49152,56);
alt_pressed = false;
}
//To release the ctl key if its pressed
if(ctl_pressed){
sendScancode(time,49152,29);
scancodeToFile(49152,29);
ctl_pressed = false;
}
logger.debug("RELEASED keychar='" + arr[i] + " char='"
+ keyChar + "'");
sendScancode(time,49152,scancode);
scancodeToFile(49152,scancode);
if(upper || setFlag){
action = 0;
debugString += "(0x"
+ Integer.toHexString(42)
+ ", "
37
+ ((action == KeyCode_FileBased.UP || action ==
KeyCode_FileBased.QUIETUP) ? "up"
: "down")
+ ((flags & KBD_FLAG_QUIET) != 0 ? " quiet" : "")
+ " at " + time + ")";
sendScancode(time,49152,42);
scancodeToFile(49152,42);
logger.debug(debugString);
}
}
}
// Checks if the keys pressed are in the combination of the alt key
public char[] altCombinationKeys(String str){
char[] arr;
if(str.substring(4).equalsIgnoreCase("Home")){
sendScancode(time, 49152, 56); // ALT
scancodeToFile(49152,56);
sendScancode(time, 0, 29); // left Ctrl
scancodeToFile(0,29);
sendScancode(time, 0, 1); // Esc
scancodeToFile(0,1);
sendScancode(time,49152,1);
scancodeToFile(49152,1);
sendScancode(time,49152,29);
scancodeToFile(49152,29);
return null;
}
else if(str.substring(4).equalsIgnoreCase("delete")){
sendScancode(time, 49152, 56); // ALT
sendScancode(time, 0, 56); // ALT
sendScancode(time, 49152, 56); // ALT
sendScancode(time, 0,
93 |
KeyCode.SCANCODE_EXTENDED); // Menu
sendScancode(time, 49152,
93 |
KeyCode.SCANCODE_EXTENDED); // Menu
scancodeToFile(0,93);
scancodeToFile(49152,93);
return null;
}
else if(str.substring(4).equalsIgnoreCase("F4")){
sendScancode(time, 0, 62); // F4
scancodeToFile(0,62);
sendScancode(time, 49152, 56); // ALT
scancodeToFile(49152,56);
38
return null;
}
else if(str.substring(4).equalsIgnoreCase("\t")){
sendScancode(time, 0, 15); // F4
scancodeToFile(0,15);
sendScancode(time, 49152, 56); // ALT
scancodeToFile(49152,56);
return null;
}
else{
arr = str.substring(4, 5).toCharArray();
return arr;
}
}
// Checks if the key pressed is a function key and then sends the scancode to the
// file
public void functionKeys(String func_str){
if(func_str.equals("F1")){
sendScancode(time,0,59);
scancodeToFile(0,59);
sendScancode(time,49152,59);
scancodeToFile(49152,59);
}
else if(func_str.equals("F2")){
sendScancode(time,0,60);
scancodeToFile(0,60);
sendScancode(time,49152,60);
scancodeToFile(49152,60);
}
else if(func_str.equals("F3")){
sendScancode(time,0,61);
scancodeToFile(0,61);
sendScancode(time,49152,61);
scancodeToFile(49152,61);
}
else if(func_str.equals("F4")){
sendScancode(time,0,62);
scancodeToFile(0,62);
sendScancode(time,49152,62);
scancodeToFile(49152,62);
}
else if(func_str.equals("F5")){
sendScancode(time,0,63);
scancodeToFile(0,63);
sendScancode(time,49152,63);
scancodeToFile(49152,63);
}
39
else if(func_str.equals("F6")){
sendScancode(time,0,64);
scancodeToFile(0,64);
sendScancode(time,49152,64);
scancodeToFile(49152,64);
}
else if(func_str.equals("F7")){
sendScancode(time,0,65);
scancodeToFile(0,65);
sendScancode(time,49152,65);
scancodeToFile(49152,65);
}
else if(func_str.equals("F8")){
sendScancode(time,0,66);
scancodeToFile(0,66);
sendScancode(time,49152,66);
scancodeToFile(49152,66);
}
else if(func_str.equals("F9")){
sendScancode(time,0,67);
scancodeToFile(0,67);
sendScancode(time,49152,67);
scancodeToFile(49152,67);
}
else if(func_str.equals("F10")){
sendScancode(time,0,68);
scancodeToFile(0,68);
sendScancode(time,49152,68);
scancodeToFile(49152,68);
}
else if(func_str.equals("F11")){
sendScancode(time,0,87);
scancodeToFile(0,87);
sendScancode(time,49152,87);
scancodeToFile(49152,87);
}
else if(func_str.equals("F12")){
sendScancode(time,0,88);
scancodeToFile(0,88);
sendScancode(time,49152,88);
scancodeToFile(49152,88);
}
}
public void scancodeToFile(int action, int scancode){
try {
40
BufferedWriter writer;
String text = action + "," + scancode;
writer = new BufferedWriter(new FileWriter("C:\\write.txt"));
contents.append(text);
contents.append(System.getProperty("line.separator"));
String aContents = contents.toString();
writer.write(aContents);
writer.close();
}
catch (FileNotFoundException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}
}
Input_Record.java
package com.hp.oo.content;
import java.awt.AWTException;
import java.awt.BorderLayout;
import java.awt.Graphics;
import java.awt.Rectangle;
import java.awt.Toolkit;
import java.awt.event.ActionListener;
import java.awt.event.KeyEvent;
import java.awt.event.MouseEvent;
import java.awt.event.MouseListener;
import java.awt.event.MouseMotionListener;
import java.awt.image.BufferedImage;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.util.Calendar;
import java.util.Date;
import java.util.Vector;
import javax.imageio.ImageIO;
import javax.swing.ImageIcon;
41
import javax.swing.JButton;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JMenu;
import javax.swing.JMenuBar;
import javax.swing.JMenuItem;
import javax.swing.JPanel;
import javax.swing.KeyStroke;
import javax.swing.event.MouseInputAdapter;
import net.propero.rdp.Input;
import net.propero.rdp.Options;
import net.propero.rdp.RdesktopCanvas;
import net.propero.rdp.RdesktopFrame_Localised;
import net.propero.rdp.Rdp;
import net.propero.rdp.WrappedImage;
import net.propero.rdp.keymapping.KeyCode;
import net.propero.rdp.keymapping.KeyCode_FileBased;
import org.apache.log4j.Level;
/**
* @author Lakshmi
* Records user actions to a file
* Scans each key stroke to determine its scan code and saves it to a file.
*/
public class Input_Record extends Input {
String myString;
BufferedImage bi_local;
JFrame diag = null;
StringBuilder contents = new StringBuilder();
Date prevTime = getCurrentTime();
Date curTime = getCurrentTime();
File[] nFiles = new File("C:/temp/screenshots2/").listFiles();
public Input_Record(RdesktopCanvas c, Rdp r, KeyCode_FileBased k, String fname,
BufferedImage bi){
super(c, r, k);
this.myString = fname;
this.bi_local = bi;
for (File file : nFiles) {
file.delete();
}
File yourFile = new File(fname);
yourFile.delete();
File yourNewFile = new File(fname);
42
try {
yourNewFile.createNewFile();
} catch (IOException e) {
e.printStackTrace();
}
}
// Sends the key presses to the remote desktop
public void sendKeyPresses(String pressSequence) {
try {
String debugString = "Sending keypresses: ";
for (int i = 0; i < pressSequence.length(); i += 2) {
int scancode = (int) pressSequence.charAt(i);
int action = (int) pressSequence.charAt(i + 1);
int flags = 0;
if (action == KeyCode_FileBased.UP)
flags = RDP_KEYRELEASE;
else if (action == KeyCode_FileBased.DOWN)
flags = RDP_KEYPRESS;
else if (action == KeyCode_FileBased.QUIETUP)
flags = RDP_KEYRELEASE | KBD_FLAG_QUIET;
else if (action == KeyCode_FileBased.QUIETDOWN)
flags = RDP_KEYPRESS | KBD_FLAG_QUIET;
long t = getTime();
debugString += "(0x"
+ Integer.toHexString(scancode)
+ ", "
+ ((action == KeyCode_FileBased.UP || action
== KeyCode_FileBased.QUIETUP) ? "up"
: "down")
+ ((flags & KBD_FLAG_QUIET) != 0 ? "
quiet" : "")
+ " at " + t + ")";
// Checks if the key F8 is pressed and then captures and
//displays the screen shot of the remote desktop
if(scancode == 0x42 && diag == null){
diag = new wizard("Screenshot", bi_local);
}
else if (scancode == 0x42) {
diag.show();
}
43
sendScancode(t, flags, scancode);
prevTime = curTime;
curTime = getCurrentTime();
scancodeToFile(prevTime, curTime, flags, scancode);
}
if (pressSequence.length() > 0)
logger.debug(debugString);
} catch (Exception ex) {
return;
}
}
// Writes the scancodes of the keystrokes to file
public void scancodeToFile(Date prev, Date cur, int action, int scancode){
try {
long diff = cur.getTime() - prev.getTime();
BufferedWriter writer;
String text = diff + "," + action + "," + scancode;
writer = new BufferedWriter(new FileWriter(this.myString));
contents.append(text);
contents.append(System.getProperty("line.separator"));
String aContents = contents.toString();
writer.write(aContents);
writer.close();
}
catch (FileNotFoundException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}
// Gets the current time to calculate the time delay between two consecutive key
// strokes.
public Date getCurrentTime(){
Calendar cal = Calendar.getInstance();
Date date = cal.getTime();
return date;
}
44
//Gets the most recent snapshot of the remote desktop
public JLabel getJLabel() throws IOException, AWTException {
long lastMod = Long.MIN_VALUE;
File choise = null;
for (File file : nFiles) {
if (file.lastModified() > lastMod) {
choise = file;
lastMod = file.lastModified();
}
}
File last = new File(choise.toString());
BufferedImage image = ImageIO.read(last);
if (image == null) { return null; }
ImageIcon icon = new ImageIcon(image);
return new JLabel(icon);
// no image available
}
}
Input_play.java
package com.hp.oo.content;
import java.awt.event.KeyEvent;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.Scanner;
import net.propero.rdp.Input;
import net.propero.rdp.RdesktopCanvas;
import net.propero.rdp.Rdp;
import net.propero.rdp.keymapping.KeyCode_FileBased;
import com.hp.oo.content.Input_Local.MyThread;
/**
* @author Lakshmi
* Play back recorded user actions
* Reads each scan code, key status (pressed or released) and the time delay from the file
* and sends it to the remote desktop in the same order.
*/
public class Input_Play extends Input {
String flag = null;
45
String str;
BufferedImage bi_local;
long t = getTime();
public Input_Play(RdesktopCanvas c, Rdp r, KeyCode_FileBased k, String fname,
BufferedImage bi){
super(c,r,k);
this.str = fname;
this.bi_local = bi;
}
class MyThread extends Thread{
public MyThread(){
}
public void run(){
File myfile = new File(str);
MyThread th = new MyThread();
Scanner scanner;
try {
scanner = new Scanner(myfile);
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
String[] tokens = line.split(",");
int time = Integer.parseInt(tokens[0]);
int act = Integer.parseInt(tokens[1]);
int code = Integer.parseInt(tokens[2]);
try {
th.sleep(time);
if(code == 0x42){
new wizard("Screenshot", bi_local);
}
else
sendScancode(t, act, code);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
scanner.close();
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
46
e1.printStackTrace();
}
}
}
// This sends a signal that the connection is ready to send messages. Also checks if only
// one thread is running to play back the recorded actions.
public void triggerReadyToSend() {
Thread mythread = new MyThread();
if(flag != "done")
{
flag = "done";
mythread.start();
}
}
}
Compare.java
package com.hp.oo.content;
import java.awt.Image;
import java.awt.Rectangle;
import java.awt.Toolkit;
import java.awt.image.BufferedImage;
import java.awt.image.PixelGrabber;
import java.io.File;
import java.io.IOException;
import javax.imageio.ImageIO;
import javax.swing.JOptionPane;
/**
* @author Lakshmi
* Compare two images based on the co-ordinates pixel by pixel and then return the
* response if the two images are same or not.
*/
public class test {
static void processImage(Rectangle r, Image img, Image image) throws IOException {
String inFile = "C:/temp/screenshots1/screenCapture_6.PNG";
String inFile2 = "C:/temp/screenshots1/screenCapture_6.PNG";
47
try {
PixelGrabber grabber = new PixelGrabber(image, r.x, r.y, r.width,
r.height, false);
PixelGrabber grabber2 = new PixelGrabber(img, r.x, r.y, r.width,
r.height, false);
BufferedImage bi = new BufferedImage(800, 600,
BufferedImage.TYPE_INT_RGB);
int[] data = null;
int[] data2 = null;
if (grabber.grabPixels()){
data = (int[]) grabber.getPixels();
//save a part of the png image
BufferedImage b = ImageIO.read(new
File("C:/temp/screenshots2/screenCapture_6.PNG"));
BufferedImage bm = b.getSubimage(20, 20, 400, 400);
String fileNameToSaveTo = "C:/temp/screenCapture_" +
".PNG";
writeImage(bm, fileNameToSaveTo, "PNG");
}
if (grabber2.grabPixels()){
data2 = (int[]) grabber2.getPixels();
}
if(java.util.Arrays.equals(data, data2) )
JOptionPane.showMessageDialog(null, "The Same");
else
JOptionPane.showMessageDialog(null, "Not the same");
}
catch (InterruptedException e1) {
e1.printStackTrace();
}
}
public static void writeImage(BufferedImage img, String fileLocation, String extension)
{
try
{
BufferedImage bi = img;
File outputfile = new File(fileLocation);
ImageIO.write(bi, extension, outputfile);
}
48
catch (IOException e) {
e.printStackTrace();
}
}
}
49
BIBLIOGRAPHY
[1] Remote Desktop Protocol (RDP), [Online]
Available:
http://etutorials.org/Microsoft+Products/microsoft+windows+server+2003+terminal+serv
ices/Chapter+3+Communication+Protocols+and+Thin+Clients/Remote+Desktop+Protoc
ol+RDP/
[2] properJavaRDP, [Online]
Available: http://properjavardp.sourceforge.net/
[3] Comparison of Java Remote Desktop projects, [Online]
Available: http://en.wikipedia.org/wiki/Comparison_of_Java_Remote_Desktop_projects
[4] Remote Desktop Protocol, [Online]
Available: http://msdn.microsoft.com/enus/library/windows/desktop/aa383015(v=vs.85).aspx
[5] Java Remote Desktop Administration, [Online]
Available: http://www.codeproject.com/KB/IP/RemoteAdminJava.aspx
[6] RDP Security – Designing Terminal Server Security, [Online]
Available: http://www.petri.co.il/securing-rdp-remote-desktop-and-terminal-serverconnections.htm
[7] WS2008: Remote Desktop Connection Architecture, [Online]
Available: http://blogs.technet.com/b/askperf/archive/2008/02/19/ws2008-remotedesktop-connection-architecture.aspx
[8] Thin Client Architecture, [Online]
50
Available: http://msdn.microsoft.com/en-us/library/ee480537(v=winembedded.60).aspx
[9] Everyday is a new day, [Online]
Available: http://zhoupengylx.blogspot.com/
[10] rdesktop, [Online]
Available: http://en.wikipedia.org/wiki/Rdesktop
[11] Elusiva - Remote Desktop Client, [Online]
Available: http://www.elusiva.com/products/RemoteDesktopClient.aspx
[12] Internet socket, [Online]
Available: http://en.wikipedia.org/wiki/Internet_socket