Verslag
Final report
2-2-2006
The development of a server that can deliver pictures of a remote operable electron microscope to multiple clients in real-time.
Graduation report Software Engineering of
N. Wesseling, January 2005
In assignment of FEI Company & FOM
Author
N. Wesseling ( 1047965 )
Supervisors
Ir. H.J.A.M. Geers ( Computer science )
Prof. Dr. H. W. Zandbergen ( Applied Sciences )
Ir. A. van Balen ( R&D )
Publisher
Technische Universiteit Delft,
Faculteit der Technische Wiskunde en Informatica,
Vakgroep Technische Informatica of Informatiesystemen
Graduation data
Spreker/Speaker/Lecturer: N. Wesseling
Onderwerp/Subject: Development of a server that can deliver pictures of a remote operable electron microscope to multiple clients in real-time.
Datum/Date:
Tijd/Time:
Plaats/Location:
Afstudeercommissie/
GraduationCommision:
3 February 2006 ??
??
??
Prof. dr. A. van Deursen
Ir. F. Ververs
Dr. ir. C.A.P.S. van der Mast
Ir. H.J. Geers
Ir. A. van Balen
Samenvatting/Abstract: FEI company designs, manufactures, markets and services products based on focused charged particle technology such as electron microscopes. One of the current research projects of FEI is an investigation of the possibility to operate an electron microscope from a distant location. This would offer a number of advantages to conventional use of a microscope.
The new technology however requires transportation of a vast amount of data over Internet from the cameras of the microsope to the computer of the User. Since the two videos, RT and Fastscan CCD are needed in real-time, fast compression and efficient transport of the data is needed. In the graduation project a server has been developed that is capable of providing real-time transport while maintaining a high video quality.
Entryword:
Type of organisation:
Research area:
Nature of research:
Applied software:
Applied methodology:
Research relations:
Follow-up research:
Real-time, transport, Internet, RT, Fastscan CCD, video quality
Manufacturer and developer of nanotechnology.
Real-time transport over networks and Internet.
Development of server, experimental.
Microsoft Visual Studio .Net 2003, JUDE
Waterfall
Vakgroep, Nanoscience, Information and Communication
Theory Group, Virtueel Lab Nederland
None
Preface
This thesis is the final report of the graduation project in the specialisation software engineering under the master computer science of the study technical informatics at
TUDelft. The project was carried out in assignment of FEI. The goal of this project was the development of a streaming server that can deliver video from a remotely operable electron microscope in real-time to clients . Throughout this project I have had support and assistance of a great number of kind people of whom I’d like to thank the following persons in particular.
I would especially like to thank Ir. H.J.A.M. Geers for supervising the project. The documents improved with his comments. Also his advice, guidance and ideas throughout this project have been useful.
Thanks also go out to Ir. A. van Balen. Not only for giving me the opportunity and means to graduate but also for making time available to comment on documents and answer questions.
I also thank Prof. dr. H. W. Zandbergen, Ir. M. Freese and Dr. J. Jansen in particular and others in general for answering any questions I had and the guidance and suggestions of H.W. Zandbergen in particular.
I appreciate the assistance that Dr. ir. RL. Lagendijk gave on the subject of compression.
The help was of significant use during the research on the effects of compression on microscopic pictures.
And finally I would like to thank friends and family for assisting me in completing this report.
Of course thanks also goes out to all those who have helped me in this project but are not listed here.
Summary
Outlay
Introduction
FEI company is one of the leading developers of electron microscopes. One of the current research projects of FEI is an investigation to the feasibility of remote control over
Internet of an electron microscope.
This would offer several economic and technical advantages; the microscope is more cost effective and less dependent on location. The location independency allows the microscope to be placed at a more favourable area with less interference producing more clear pictures. The microscope can also be hired by clients and shared by universities. So nanoscale research can be performed with a higher accuracy making new discoveries possible.
In the graduation project a streaming server is designed and implemented that deliver visual data from the microscope to the user in real-time. The data are delivered over
Internet to clients at remote places.
The graduation project consists of two parts, a literature study and a development project. The literature study investigated compression algorithms and protocols to find the most appropriate compression and communication techniques. The development project was the practical part of the graduation project and was directed at the development of the server.
Project context
The microscope used in this project is a version of the Tecnai TEM series produced by
FEI. There are three types of video pictures generated with this microscope, Fastscan
CCD (FCCD), Slowscan CCD (SCCD) and RT. Pictures of FCCD and RT are used to control the microscope and should therefore arrive in real-time at the client, SCCD is used for analysis and has to be of high visual quality.
The bandwidth of the Internet connection with customers can range from 1 Mb/s up to
1 Gb/s. The actual bandwidth is unpredictable as it is based on the ‘best effort’ IP protocol. This behavior has to be coped with in the project.
Planning
A two-phased plan was followed in the development project. In the first phase the basis of the server was developed, that would be improved on in the second phase by the feedback of users. During both phases an analysis of the system requirements document
(RAD), a global design document (SDD) and a detailed design document (ODD) were to
be produced. Due to a lack of time this plan was abandoned for a different planning that focused more on the implementation of server and the codec.
Execution
Research
The client requested more attention to the aspect of compression so an additional research project was carried out. Three aspects were investigated:
The subjective evaluation of users of pictures compressed at various ratios.
The nature of the noise present in the microscope pictures and the filtering methods.
The mean and variation of the pixel values of a picture.
In the first part of the research several fastscan CCD pictures were compressed with
JPEG and JPEG 2000 at ratios from 0.1 up to .5 bpp. These pictures were then shown in a random layout to five users of an electron microscope who were asked to select the pictures that were still acceptable and those that were not and to explain the selection.
This lead to the conclusion that the pictures could be compressed significantly without losing to much visual quality. Pictures compressed with JPEG2000 at .1 bpp were still acceptable; pictures compressed with JPEG were considered good enough at .3 bpp.
To obtain even better compression ratios the research also focused on the nature of noise in the CCD pictures and possible methods of removal. If some noise is removed the compression ratio can be increased without degrading the visual quality of the pictures further.
FCCD and SCCD pictures contain different types of noise such as bias noise, dark current noise, X-ray noise and Poisson noise. The default method of filtering bias noise and dark current noise are simple, efficient and effective. The effect of X-ray noise on data size is in most cases insignificant. Poisson noise, which is caused by the unpredictability of the number of electrons that fall in a certain interval on a pixel, can be reduced with a Gaussian filter. An application was created to test the effect of such a filter. Some gain in signal to noise ratio was achieved but as the improvement was not significant further experimentation was cancelled.
The last part of the research, that analysed the statistical properties of the data, indicated that the variation of the pixel values is quite small and that most values lie around the mean. By using values relative to the mean instead of absolute values the number of bits required per pixel is reduced. This would allow more data to be sent per second. But it turned out that pictures preprocessed to contain relative values and compressed with
JPEG were larger in size than directly compressed pictures. So preprocessing the data in this manner cannot be used to reduce the data size.
Analysis
The development project started with an analysis of the system requirements. The analysis concluded that efficiency is the main criterion of the system. The system also has to be able to cope with various bandwidths so flexibility in data transport is important as well. To allow the system to be extended and improved in the future it also should be easy to modify.
Design
In the design the main part of the system manages the connection, monitors the performance and is able to change the compression ratio if necessary. The actual data transport has been separated from the management of the data transport. This allows the system to change the configuration of the data transport during run-time. Multiple clients can share a single data stream. In the design the server is responsible for the management of the data transport. The client reacts passively to changes by the server.
The server and client start and end transmissions with the session protocol RTSP. A class at both the client and server is responsible for handling the exchange of messages in this protocol. The RTP (Real-time Transmission Protocol) protocol is used for the data transport, RTCP (Real-time Control Protocol) for the exchange of statistics concerning the data transport. More details regarding RTP and RTCP can be found in the thesis report “Real-time transport of microscope video over Internet”.
The four main classes in the design are ClientManager, DataStream, DataStreamList and
TaskScheduler. DataStream represents a datastream from the server to one or more clients. ClientManager represents a single client and manages all transmissions from the server to that client. A ClientManager can create, add and remove DataStream objects from the DataStreamList class that keeps track of all the datastreams.
TaskScheduler is responsible for scheduling the activity of datastreams and clients. This class activates the datastreams successively. Once activated the datastream sends frames to all clients to which it is connected. The TaskScheduler also periodically allows the
ClientManagers to communicate with their clients.
Implementation
At the start of the implementation different codecs libraries and data transport toolkits were evaluated. Efficiency and portability were the main criteria. The object oriented
C++ language was selected as programming language for its efficiency and widespread use.
A total of eleven compression formats have been considered, the well known JPEG format, but also the more exotic SPIHT and COMPFITS. The most suitable libraries of this selection are the commercial codec Kakadu (JPEG2000) and open source IJG. Both provide efficiency and portability. IJG was chosen as the first codec to be used in the implementation because it offers a better overall speed.
Most libraries and tools for data transport use Winsock, which is based on Windows and is therefore not portable enough. The ACE Toolkit is a better suitable tool because of its portability and efficiency and is therefore used for the data transport. Open source
RTP and RTCP classes of Vovida are used for the handling of the exchange of RTP and
RTCP packets.
The implementation is based on the ‘pipe/filter’ paradigm. Stream classes derived from the ACE class Stream are used as pipes. Task classes derived from the ACE class Task are used as filters. Each Task class specializes in a certain functionality that has to be applied on the data, such as memory retrieval/storage, compression/decompression, packing/unpacking and transmission/reception. Data flows through the stream classes along all the tasks on the stream.
During the implementation problems with the tools slowed down the progress. It also took some time to learn to use the ACE toolkit and IJG library. Difficulties with the system memory storage also formed a problem that was hard to locate and solve.
Testing
After the implementation was finished the system was tested on both performance and stability. Different pictures were send across the Internet to a client and were evaluated on quality and order of arrival. To measure the stability of the server and client they were run for 3 hours.
After these operational tests the performance was measured in terms of framerate, jitter and number of packets lost. The tests were run with different compression ratios, number of data streams and fragmentation of pictures. The pictures were sent to a local computer, over a network and over the Internet.
All the tests were passed satisfactory. A framerate of 26fps was achieved for a bandwidth of 12.5 MB/s for a compression ratio of 0.5bpp for two streams.
The testing of server and client was problematic due to the lack of a good test set-up and dependency on people from outside of the project. In a number of tests firewalls blocked incoming data. In other tests routers, insufficient bandwidth, processing power, memory or the Operating System caused problems. Due to these problems the testing took several weeks instead of one.
Evaluation
Project evaluation
One problem during this project was the discrepancy between the project priorities and expectations of the client. Another problem were the delays throughout the project. The first problem could have been avoided if more frequent communication with the client and users had occurred and if a more assertive style of communication would have been applied. Three possible reasons for the delays in the project are:
Lack of experience with the tools used
To much focus on documentation
Underestimation of the time required.
Because of the delays there was not enough time available to complete all parts of the system. Aside from completion of these parts the system would also benefit from implementation of the Resource Reservation Protocol (RSVP). It needs more precise error control and some form of resource control to prevent it from consuming the entire bandwidth. These improvements of the server and client can be implemented in a future project by FEI.
Despite the delays the system has achieved the main goal of efficient transport of visual data. Also the well structured code and extensive documentation will make it easy to modify and extend the server and client.
Contents
1. Introduction
1.1. Introduction
Since its invention in 1931 the electron microscope has contributed to numerous breakthroughs in different scientific areas such as medical and material science. Electron microscopes are the successor of optical microscopes using electron bundles instead of light the limitations of light are overcome. This allows scientist to see subjects on a nanoscale level such as the interior of cells.
Fig 1.1: View of a flea with an electron microscope
FEI company, among the top producers of electron microscopes, is now bringing the technology to the next step. In one of their investigations the possibilities of remote control of an electron microscope are researched [1] .
Remote control would offer several economic and technical advantages. Clients can hire such a microscope instead of buying one which attracts new and more clients.
1
Another advantage is that clients will be able to operate this microscope from any place on the world. The microscope can be placed at a more suitable locations where there is less interference from radiation, vibrations etc. To TUDelft and other universities that cooperate with FEI in VLN (Virtual Lab Netherlands) the main advantage is that microscopes can be shared which reduces costs of the research.
A prototype will demonstrate customers the benefits of a remotely controlled microscope.
At this moment most software components of the system are in development. For the transport of visual data between the computer of the microscope (server) and the computer of the user (client) code has been designed and implemented. The implementation however is to slow for practical purposes and a faster, more efficient transport system is required. The goal of the graduation project is to design and implement a streaming server that delivers visual data from the cameras to the client in real-time.
The graduation project consists of two parts, a literature study and a development project.
The literature study investigated compression algorithms and protocols. It concluded that MPEG is the most suitable codec for low bandwidths while the less complex JPEG is more suitable for higher bandwidths. Regarding the transport of data the study came to the conclusion that the RTP (Real-time Transmission Protocol) protocol on top of UDP is the best choice with the RTCP (Real-time Control Protocol) protocol for the exchange of data transport statistics between server and client. See the thesis of the literature study,
“Real-time transmission of microscope video over Internet”, for further details.
The development project was the practical part of the graduation project and evolved around the development of the server. This thesis discusses the development project and concludes the graduation project.
1.2. Report Structure
Chapter 2 discusses the planning of the project and the approach that was taken in the project.
Chapter 3 will give a short description of FEI company.
The context of the project will be given in chapter 4, which will treat the prototype electron microscope, the different video types the data transport and quality properties of the video and transport.
Chapter 5 discusses the research of the visual data that was carried out during the project. This research focused on the properties, effect of compression and noise filtering of the data.
The next three chapters contain the main parts of the documents that were produced in the three phases of the development project, analysis, global design and detailed design. The analysis of the system requirements can be found in chapter 6, the global design of the client and server in chapter 7 and the detailed design in chapter 8. Chapter 9 discusses the implementation of the server, the rationale for the tools and libraries that were chosen and the overall structure of the code. After completion of the server and client several tests
2
were run to measure their performance and test the stability. The findings of these tests are discussed in chapter 10.
The last chapter of this report, chapter 11 conclusion, presents an evaluation of the entire graduation project, which reflects on the mistakes made, successes achieved and gives advice concerning further development of the server and client.
Appendix A contains the classes and sequence diagrams of the detailed design that were not included in chapter 8. Appendix B contains the classes of the implementation that were not included in chapter 9.
3
2. Planning
General
The development project was planned to start in January 2005 and finish in September
2005. Because the literature study required more time than planned, the development project was delayed to February 2005. During the development project more delay was encountered and the original planning was altered. Both plannings are discussed.
First planning
Table 2.1 shows the original planning for the development project.
Table 2.1 The original planning of the development project
Activity
Planning
Start date End date # Weeks
PHASE I
31-1-2005 18-2-2005 3 Analysis (RAD)
Code monitoring system* 21-2
Presentation of thesis report 7-3
4-3
11-3
2
1
Global design (SDD)
Detailed design (ODD)
Implementation
Testing
14-3
28-3
11-4
2-5
25-3
8-4
29-4
6-5
2
2
3
1
Revision of RAD
Revision of SDD
Revision of ODD
Implementation
Testing
Creation of final report
Presentation
Holiday
PHASE II
9-5
16-5-
23-5
30-5
20-6
27-6
11-7
18-7
13-5
20-5
27-5
17-6
24-6
8-7
15-7
12-8
1
1
1
3
1
2
1
4
Marge 15-8 30-9 6
* A pplication to add noise to a picture and to measure the signal/noise ratio for two pictures. This application was requested for by the client.
Project time was divided in two phases. In the first phase the most important functionality was to be implemented. In the second phase non-essential but desired functionality would be developed. The reason for this is twofold. It would allow users to give comments on the results of the first phase. That feedback could then be used in the second phase. The second reason is that the exact functionality to be implemented during the second phase
4
could be decided on in a later stage of the project. Therefore it added some flexibility in the project.
In the first phase the main focus was on the development of code for the video transmission. The following steps were planned in that phase.
1.
Analysis
In this stage the requirements of the system in terms of functionality (what must the system do) and quality (how good should the system be) was to be discussed with users and the client. These findings were to be reported in the Requirements Analysis
Document (RAD). The purpose of this is to describe how the system should behave and to create consensus with users.
2.
Design
The structure of the system had to be laid out in the design stage. First an overall global design had to be made in which the main modules of the system would be designed. Then a more detailed design would be made. The global design is described in the System Design Document (SDD) and entails the system structure for both phase 1 and 2. The Object Design Document (ODD) contains the more detailed design. The SDD and ODD would be presented to the users and client. With their comments these documents were then to be revised.
3: Implementation
Then the design the system for phase 1 would be implemented. After, but also during, the implementation, the code would be tested for stability and performance.
The next phase was planned to begin with a brief revision of the RAD using comments on the results of the first phase. The SDD also would be revised if necessary. Next the
ODD for phase 2 would have been written and phase 2 implemented. The system would then be tested to eliminate most bugs.
The client and server were to be completed and tested before the second week of
July/August. The final report of the whole project could then be written in the following two weeks while the presentation could be done in the remaining week of July/August.
The month September was used as a marge to allow unexpected delays during the project to occur.
Second planning
During the project most steps took longer than estimated. To compensate for the delay the planning was altered. In the end of May the client indicated that he wanted more focus on
5
the aspect of compression and noise in data. This required a different approach in the project and which was processed in the new planning.
Table 2.2 shows the new planning
Table 2.2 New planning
Activity
New planning
Start date End date # Weeks
Research of data
Implementation of Codec
Testing of Codec 27-6
Improvement of Codec 4-7
Implementation of Server 11-7
Testing server
PHASE I
30-5-2005 10-6
13-6 24-6
1-7
1-8
8-7
29-7
10-8
Holidays 11-8 2-9
2
2
1
1
3
2
4
RAD Revision
SDD Revision
ODD Revision
Improvement of Server and
Codec
Testing of Codec
PHASE II
12-9
19-9
26-9
3-10
Testing of Server
Creation of final report
Presentation
Marge
17-10
24-10
7-11
28-11
5-12
16-9
23-9
30-9
14-10
21-10
4-11
25-11
1
1
1
2
1
2
3
2-12
30-12
1
4
To guarantee that the vital parts of the server and client would be complete further work on the ODD document was delayed. Instead implementation of server and codec was started. The codec was to be implemented and tested first, then the implementation and testing of the server would start.
The next phase was similar to the second phase of the original planning. In this phase the server and client would be improved One to two weeks were available for changes to each document. Two up to four weeks for extending and improving the server.
Because the implementation of the server and client took more time than planned phase 2 has not been executed.
6
3. FEI Company
FEI was founded in 1971 to produce crystal materials for field emission research. The
World Headquarters Campus of FEI is located in Hillsboro, Oregon, USA. Over 1700 employees worldwide work for FEI.
The company is specialized in the development of focused electron and ion beams technology. With this technology materials can be visualized and analyzed on nanoscale levels.. The technology is also used to modify materials on nanoscale level [2] .
In 1997 FEI merged with Philips Electron Optics, a company located in Eindhoven, the
Netherlands. Philips Electron Optics was the first company in the world to produce commercial electron microscopes. Their commercial electron microscope was introduced in 1949. Products that Philips produced before the merger are now manufactured as part of the product line of FEI. The scanning electron microscope (SEM), the Transmission electron microscopes (TEM), the Stylus NanoProfilometer (SNP) and the DualBeam™/
Fab-Based are some of the products that are manufactured and sold by FEI [2] .
Fig 3.1: Corner cross-section of multi-layer material using focused ion beam. Sample courtesy of
FEI Company Applications Laboratory.
7
4. Project context
This chapter contains parts of chapter 2 of the thesis report, “System requirements”.
To accommodate to a wide range of customers the server has be flexible enough to handle a diverse range of platforms and connections. Section “Problem scope” discusses the type of microscope that is used and the properties of the video and data transport. The most important requirements of the system are described under quality properties.
4.1. Problem scope
General
Optical microscopes have a maximum magnification of 1000x. This is a consequence of the diffraction of the wavelength of light. In 1931 Max Knott and Ernst Ruska in 1931 introduced the electron microscope that uses an electron beam instead of light. Electrons have a much smaller wavelength than light and can therefore magnify up to nanometers.
Magnets are used to form lenses that focus the electron beams at a subject. With cameras such as CCD (Coupled Charge Device), at the end of the microscope, the electrons are registered so that the subject can be visualized [3] .
Fig 4.1: The electron microscope uses magnets and electrons instead of light and lenses
8
The microscope used in the prototype of this project is a version of the Tecnai TEM series of FEI. This is a highly automated microscope using 300 kV that can magnify up to nanoscale levels.
Fig 4.2: The Tecnai microscope
Video
There are three main kind of images that have to displayed at the client, Real-Time(RT),
Fastscan CCD(FCCD) and Slowscan CCD (SCCD). Slow- and fastscan CCD video are formed from a series of separately taken pictures with a CCD camera. RT video is captured with a camera focused on a screen at the microscope. FCCD and SCCD contain different types of noise.
The fastscan CCD and RT video are used to set the parameters of the microscope and for moving, scaling and rotating the sample to find a good view of some part of the sample.
They should therefore be shown to the user in real-time.
A slowscan CCD picture takes longer to make and contains more detail with less noise.
These pictures are used to analyse the sample. The most important criteria for transmission of a slowscan CCD picture is the visual quality of the data. This type of data does not have to arrive in real-time at the client but has to keep its high resolution quality.
Table 4.1: Video properties
9
Video properties
RT FCCD
Resolution 512x512 512x512
#bits per pixel 8 bit 12 bit
Max. latency 50 msec 50 msec
Amount of noise
Noisy Noisy
SCCD
1024x1024
16 bit
250 msec
Little noise
Internet
The client and server communicate over Internet. This makes it possible to connect with the microscope from any location. It also gives specific problems that need to be taken care of.
Internet is at this moment based on the IPv4 protocol (IPv6 is nearing deployment). The
IP protocol allows different types of networks to interact, creating an inter-protocol network in effect. Internet is the largest existing inter-protocol network.
IP works on a ‘best effort’ approach. Best effort stands for sending data distributed over several packets only once without control on the correct arrival of the data. Packets can be lost, duplicated, altered and arrive out of order. This makes delivery of data over
Internet unpredictable and unreliable. Also no guarantee can be made for a constant available bandwidth. Latency, jitter and video quality in general are difficult to control when the circumstances are volatile. Internet is therefore a problematic medium to deliver live video [4] .
Bandwidth
The bandwidth of a connection is the amount of data that can be send per second over the connection. University networks normally have a high bandwidth, In the building where the prototype for instance is connected to a gigabit connection. When multiple users share these networks, no assumption of a specific bandwidth can be made, but throughput is probably high enough for the amount of data that needs to be send. External companies however may not have such high bandwidth available for economic reasons. The slowest connection in the connection chain determines the overall speed. Therefore the system should be able to change its encoding strategy according to the available throughput.
Bandwidths ranging from 1 Mb/s to 1 Gb/s will be taken into consideration.
4.2 Quality properties
Video quality
Video quality consists of the resolution (pixels per picture) and the monochrome variation (greyscale). Another determinant of video quality is the amount of noise that is present in the image. Noise is random disturbance in the image that holds no visual information.
10
The main videos, used to operate the microscope, RT and FCCD, contain noise. The noise can be subdivided into different kinds. Transmission errors, compression and decompression add extra noise to the images. In an ideal transmission there would be no extra noise. If the added noise is limited the user won’t notice the influence.
Latency
Latency is the delay between the moment of image capture by the camera of the microscope and the moment of display of the images. When a network connects the microscope and computer, the processing time on the server and client and the transmission time of the video over Internet add up to the total latency.
A user of a remote controlled microscope should have the feeling of direct control of the microscope. Otherwise operating the microscope will give an unnatural experience that complicates the control and creates resentment among users. For a direct response the latency has to be below 0.05 sec for RT and FCCD video and below 0.25 sec for
SCCD video.
Jitter
Where latency is the delay, jitter is the variation of the delay. In practice a high jitter means that the system might react instantly at one moment and very slow the next moment. The user will perceive a system with much jitter as unpredictive and annoying.
Low jitter is therefore preferred.
Fig 4.2: The upper image shows a graph with a low latency but a high jitter, the lower image has a higher average latency, but the jitter is low.
Frame rate
The frame rate is the number of frames (images) shown per second. When the frame rate is above 15 fps the video is seen as continuous motion instead of a sequence of separate frames. The frame rate should be at least 20fps to give the user the impression of fluent changes in frames.
11
Priority
All the previous described quality aspects of the video are important for the creation of a pleasant working environment. But restrictions in bandwidth and computational power limit the quality that can be obtained. Thus an emphasis on some quality aspects over others is required.
The choice is between direct control and visual quality. If a user does not get a response on a command within 0.05 secs, operating the microscope becomes problematic and annoying. It will be unclear to the user what the current status of the system is. When the reaction is fast, but the resolution is low, a user is able to continue his project. Latency, jitter and frame rate are more important than the video quality in case of the FCCD and
RT video.
12
5. Research
During the project the client requested a research on the properties of the visual data and the effects of compression. The following three aspects were investigated:
The subjective evaluation that users give of compressed and decompressed pictures
The nature of the noise in SCCD and FCCD data, the standard noise filtering techniques and the possible use of alternative filters.
The mean and variation of the values of the pixels in SCCD pictures.
This chapter will discuss the set-up and findings of the research.
5.1. Subjective evaluation
Several fastscan CCD pictures were compressed and decompressed with JPEG and JPEG
2000 at different ratios from 0.1 up to 0.5 bpp. These pictures were shown in a random order to five users of an electron microscope. They were asked to select the pictures that were still acceptable, those that were not and to explain the selection.
Fig 5.1 Some of the pictures that were shown to users, from left to right: the original picture, the same picture compressed with JPEG at 0.4 bpp and compressed with JPEG2000 at 0.4 bpp.
The selection that each user made corresponded to the selection of the other users, so the result that were obtained are consistent. It turned out that the FCCD pictures can be compressed significantly without a significant loss of visual quality. With JPEG2000 pictures compressed to 0.1 bits per pixel (bpp) were still considered acceptable. For JPEG the pictures below 0.3 bpp became to distorted to be useable, but from 0.3 bpp and above the pictures were evaluated as being acceptable. This result confirms a similar result of the study that was carried out in the literature study.
5.2. Noise and filters
13
The results of the subjective evaluation show promising results, but it might be possible to achieve an even higher compression ratio. The amount of noise present in pictures limits the compression ratio that can be achieved. A picture with a high noise level can not be compressed without degrading the visual quality to unacceptable levels. It was investigated how the influence of noise in the pictures can be reduced.
FCCD and RT images contain a substantial amount of noise. Different filtering tools in the microscope are able to reduce the noise to form a clear picture. The five different types of noise shall be discussed along with a description of corresponding filter. Then the possibility of an improvement of alternative and/or new filters is discussed.
Noise
Fig 5.2: On the left is an example of a noisy picture which has been filtered in the right picture.
Note these pictures are only an example.
Dark noise
The spontaneous emission of electrons is dependant on temperature. It results in a snowlike appearance on the image, called dark noise. The amount of noise will generally double for every increase of six to ten degrees C. An exposure time twice as long will create roughly twice the amount of dark noise. Dark noise scales with temperature and exposure duration.
To filter this noise a picture is made at a certain temperature and exposure duration without shutter. This picture, called dark frame, therefore only contains dark noise. The dark frame is subtracted from microscope pictures that are shot at the same temperature and exposure [5][6] .
Flatfield correction
Some pixels of the camera are more or less sensitive to electrons. So if all pixels receive the same amount of electrons the picture will have different light intensities at different pixels. The electron ray is also more intense in the center than on the edge of the screen producing variations in the pixels. This type of noise is removed by the flatfield correction procedure. A picture is made with constant exposure. The sensitivities of the
14
different pixels can be calculated from this picture. In the flatfield correction procedure the pixels of the picture are multiplied with a factor depending on the relative values of the pixels in the flat field picture. This procedure brings the sensitivity of the pixels to an equal level [5][6] .
Bias noise
Another type of noise, bias noise is formed by the inexact transformation from analog values to digital values. Prior to the analog to digital conversion the amount of charge in the camera has to be amplified to make it large enough to be read out. This amplification adds a small amount of noise to the picture even when there is no exposure. Bias noise does not depend on microscope settings and only scales with the amount of magnification that is applied on the picture.
The bias noise is removed in the same procedure as the dark noise removal. An unexposed picture, the bias frame, is made with a closed shutter. The bias frame is removed from pictures that are made with the microscope, which reduces the effect of the bias noise. The bias frame and dark frame are usually shot in a combined frame which is later subtracted from the other frames [ 5][6] .
Poisson noise
A fourth kind of noise is Poisson noise. Poisson noise is the result of the unpredictability of the number of electrons that will fall on a pixel in a certain amount of time. Some pixels may receive more or less electrons when their exposure is equal. The distribution of electrons follows the Poisson distribution for each pixel. This type noise is not removed [5][6] .
X-rays
A fifth type of noise is caused by X-rays that are a side effect of the microscope. X-rays are mostly formed at high energy intensive operations (high magnification) and create spikes in the picture.
Filters
The bias noise, dark noise and flatfield noise have a simple but fast and effective filtering method. The filtering procedure only involves subtraction and multiplication which can be done with modern computers in milliseconds. No improvement can be achieved in that area. X-ray noise does not have a significant effect on the data size so we will focus only on the filtering of Poisson noise.
Many sophisticated filter algorithms have been designed to remove Poisson noise and a lot of research is performed in this area. Most of the methods achieve a substantial improvement of the picture quality but involve complex time consuming algorithms, such as the Fourier transform filter. The filter requires about a tenth of a second on many modern computers, to long for real-time processes.
15
One method that can be applied in milliseconds is the Gaussian filter which introduces a blur effect in the picture.
Fig 5.3: A picture on pixel level
The Gaussian filter works as follows: In figure 5.3 the green pixel is currently being processed by the filter. The values of the neighbors, the red pixels, are added to the value of the green pixel through some weight. The same is performed on the other remaining pixels until all pixels have been processed.
The region in which the pixels are considered neighbours can vary. For instance in picture 5.3 only the direct neighbours, the bright red pixels contribute. In other settings pixels farther away, the dark red pixels, contribute as well. There is no upper limit to the size of the region in which the pixels can be considered neighbours except for the picture width or height. The amount of contribution that a pixel has depends on the distance to the affected pixel. Pixels that are close contribute more than pixels that are distant [7] .
A simple 2D Gaussian filter application has been created to test the effect of this method on microscope pictures. Different amounts of Poisson noise were artificially added to the pictures and filtered. The difference between the filtered pictures and unfiltered pictures was measured in root mean square error of the pixel values of two pictures.
Formula 5.1: The RMSE formula, N is the number of values, f i
is the values of the i’th pixel of the filtered picture, u i
is the value of the i’th pixel of the unfiltered picture.
RMSE
1
N i n
1
( f i
u i
)
2
The root mean square error of the pictures improved slightly after being filtered, but the difference was not substantial and did not seem to depend on the amount of noise. This might be due to the number of neighbours that have been used in the process. If more neighbours can be processed efficiently (within 5 milliseconds) the filter would be more effective. Further development on the Gaussian filter was delayed, but it might be worthwhile to pursue this subject further.
5.3. Data analysis
One of the expert users indicated that the relevant section of the data usually is only part of the entire available range. For example a picture might have 16 bits per pixel with light intensities from 0 up to 65536 available, but only contain relevant data in the range 200 to
600. By only transmitting the important part of the entire spectrum data can be
16
3500
3000
2500
2000
1500
1000
500
0 compressed. Also the values of the pixels might be prescaled relative to the mean instead of absolute if the deviation is small enough which would also reduce the amount of bits required for the values. An analysis was done to measure the mean and variation of the pixel values in a number of SCCD pictures.
The result of the analysis are shown in table 5.1 and graph 5.1. The standard deviation is usually small so most pixel values will be close to the mean of the picture. It also appears that the mean differs significantly between different pictures.
Graph 5.1: The distribution of the pixel values in a sample picture. On the y-axis are the frequencies of the values, on the x-axis the values. ptcontact004.dat
17
Fig 5.4: Ptcontact004
.
7
Because of this promising result the theory was tested in practice. Pictures were pre-processed and compressed with JPEG. The achieved compression of the pre-processed pictures was however lower than that of directly compressed pictures at the same compression settings. It turned out that when using
JPEG no extra reduction in data size can be achieved by precomputing the relative values.
8
9
10
11
12
13
14
15
4
5
6
16
17
18
19
Table 5.1: The mean and standard deviation of the pixel values
of different SCCD pictures.
Picture
#
2
Analyze results of pixel values
Mean pixel value
853
Standard deviation pixel value
78,48
3 1920 133,28
1966
1943
1661
4086
802
868
1131
1028
187,46
199,76
176,96
442,49
87,85
93,56
125,36
116,64
177,94
769
191
920
957
1440
1512
1124
1308
195,74
159,70
188,21
115,89
45,84
193,70
194,38
18
6. Requirements analysis (RAD)
This chapter contains the main parts of the Requirements Analysis Document (RAD) that reports the findings of the system analysis. During the analysis information was gathered on the current system and the requirements of the client and users for the new system.
6.1. Current System
Fig 6.1: The current system
The prototype is the first remote controlled microscope produced by FEI. The system consists of a communication system, a microscope and a UI. The system performed correctly during a trial run except for some minor problems. It has however significant delay, which is clearly visible to a user.
There are three important servers, the Acquire Server that stores data coming from the microscope, the TEM server that handles commands of the user and the Web Server that provides an User Interface to a user.
19
The Web Server provides a webservice for each possible command of the client. During the start up of the client application the list of available webservices is downloaded from the Web Server. With this list the client knows all the commands that the Web Server supports. By sending a command via the User Interface, the corresponding webservice is activated.
The TEM Server translates the user commands to specific operations of the microscope.
It is connected by a 1-gigabit optical network to the microscope. The Web Server uses the
TEM Server to control the microscope. It is vital that the TEM Server remains operational while the microscope is running.
The User Interface of the client is a Flash MX application. Video and data of the microscope is received from the Acquire Server. Macromedia is used to transmit the video. The UI is available as a web application that runs in the browser at the client computer.
The three servers operate at the same computer, the Tecnai pc. At the microscope a second computer is located which controls the hardware of the microscope directly. A 1 gigabit connection provides the link between the two computers. For the communication
TCP/IP is used.
6.2 Proposed system
This project is focused on the delivery of the transport system for visual data from the server to clients. The proposal for such a system is discussed in this section.
6.2.1. Functional requirements
The functionality that the new system must provide is listed in this section. The remote controlled microscope is a relative new technology. The desired functionality of the system is not completely defined. The functional requirements may therefore change during the project due to feedback of users.
6.2.1.1. Transmission of video *
1.
Optical data (RT and CCD video) from the microscope shall be constantly delivered to the client UI by the Acquire server.
2.
A user shall be able to start a session with a microscope. During such a session the user receives data from the microscope.
3.
A user operating the microscope (operator) must be able to end the session thus releasing control over the microscope.
20
6.2.1.2. Cooperation
1. An user operating the microscope must be able to start the cooperation
mode with other users. In this mode all users can view the data from the microscope.
Only the first user (Operator) will be able to operate the equipment.
* transmission of remote control commands is outside of the scope of this graduation project.
2. Two users shall be able to communicate by chatting and by graphical interaction
with the video presented.
3. A user shall be able to send data located at the client and at the server to other experts.
4. A user can exit a cooperation session. If the user controlling the microscope leaves the session than the cooperation mode is ended for all users.
6.2.1.3. Configuration
Each user is able to set the desired quality parameters: transport time, frame rate, pixel depth and resolution at a specific level.
6.2.2. Non-functional requirements
In this section the level of quality (quality requirements) that the system should offer is discussed.
6.2.2.1 User Interface and Human Factors
The design of the User Interface is not part of this project. The system does however affect the working conditions of a user. The main goal is to create an optimal working environment. To realize that the system will meet the following requirements.
The required properties of the data are dependent on the video type and may vary during an experiment. A user can provide the system with the desired transport time, signal-to-noise ratio, refresh frequency and pixel depth of the data. The system shall display the actual realized performance to the user.
Errors and other problems will be corrected in such a way that a user does not become aware of their existence. When however a problem is too serious or affects the working conditions of a user a message will be shown to the user.
6.2.2.2. Documentation
Extensive documentation must be present for software engineers to extend and/or modify the system. The documents will also contain the requirements. The following documents will be created during this project.
21
Requirements Analysis Document (RAD): contains the requirements for the system in terms of functionality and quality.
System Design document (SDD): contains a global design for the overall system.
Object Design Document (ODD): contains a detailed design for the system.
6.2.2.3. Performance characteristics
The provided quality is highly dependent on the available bandwidth and the configuration of the client system. The transport system will adapt to different circumstances to optimize the delivered quality/performance.
The main criteria that the system has to take care of are the transport time and the quality of the data. Some data requires less accuracy and is needed in real time, such as
RT video. Other data may take longer to transport but require more accuracy, such as slowscan CCD. The actual quality/performance trade off differs per data set. A user can set the desired level of the quality criteria.
6.2.2.4. Quality issues
The system must be accessible to a user any time during the week, as long as no other work is being performed at the microscope or server. The program shall perform correctly, which means as specified, in at least 95% of the time. The main focus will be on the correct arrival of the visual data. Errors in less important data such as RT and
FCCD video might not be corrected to speed up the transfer.
6.2.2.5. System modifications
When the prototype is developed further into a complete version, functionality may be added, removed or altered. To make modifications to the transport system more easy clear documentation on the different project phases, will be provided. Structured coding will help in making the program understandable. Furthermore the application will allow the compression codecs to be switched without changing the program.
6.2.2.6. Security issues
The research the user is working on may generate sensitive information of some company. Other companies might gain an advantage by obtaining their data. Security is an important aspect of the final system. The task of making the system secure is however not part of this project, no security measures will be implemented.
22
6.2.3. System models
6.2.3.1. Use cases
In this section a few use cases for the new system are listed. The Operator in these use cases is the person that is remotely operating the microscope and receives video from the microscope. The Observers are those who receive video and data from the microscope but do have not control over the microscope.
Use case name VideoTransfer
Participating actor(s)
Entry condition
Flow of events
Exit condition
Special requirement
Initiated by Microscope
Communicates with Operator, Observers
1: The microscope has started capturing video with the RT and CCD cameras.
2: The Acquire Server sends all the videos to the client of the Operator. And if they are present to the clients of the
Observers as well.
3: When the client receives the video it is displayed to the
User.
4: When the session ends the video transfer is stopped.
The RT video shall be displayed within 0.020 sec after it is captured.
Use case name
Participating actor(s)
Entry condition
Flow of events
Exit condition
Gather Analysis data
Initiated by Operator
1: The Operator/Observer using the Analytical screen of the UI instructs the microscope to obtain data for analysis.
2: The microscope performs the required actions, the
Acquire Server sends the gathered data to the client.
During this process a progress bar is shown to the
Operator/Observer to show the status of the procedure.
3: When the procedure is completed the
Operator/Observer is informed, he/she may access the data at the client.
23
Use case name
Participating actor(s)
Entry condition
Flow of events
Exit condition
6.2.3.2. Object model
6.2.3.2.1. Data dictionary
Fig 6.2: The different data types
StartCooperation
Initiated by Operator
Communicates with Observer
1: The Operator sends the Observer an invitation via email to take part in the session.
2: The Observer accepts the invitation by clicking the url in the email.
3a: The client of the Operator is informed that the
Observer joined the session and shows a screen for cooperation.
3b: The microscope will send data that the Operator receives to the Observer as well. The preferences of each
Observer (resolution, video quality, etc) are uphold by the server.
The microscope captures all the data that is of concern. Data can be divided into visual data and graphs that the microscope collects. The X-ray energy dispersive spectrum is an example of one of these graphs.
The visual data, requiring more space for storage and transmission are the most important. The properties of these data are highly variable depending on many factors, such as the type of sample used. There are three different types of images, RT video, slowscan CCD and fastscan CCD.
24
7. Global design (SDD)
This chapter contains the main parts of the System Design Document (SDD). The SDD encompasses the global design of the client and server.
7.1. General
7.1.1. Design Goals
Performance
The main design goal of this project is the performance of the transport system. In case of real-time data such as FCCD video the system will have to focus on the transport time while for other data the quality of the data is the main concern. The system must provide the desired performance for all the properties of the data that the user may set.
Reliability
Error handling
The transport system is part of a prototype; the tolerance to errors is not the main concern. The system shall however correct errors in the data that are introduced during transport. Other error handling is not implemented.
A log file is stored on disk at client and server for each session. The log file records the status of the transport system, communication between client and server (or client and client) and the progress of the data transport. The level of detail that is maintained can be set prior to the session. At the lowest logging level no log file is recorded, at the highest level very detailed information for debugging is recorded.
Maintenance
System structure
When the present prototype is developed further into a complete system, functionality may be added, removed or altered. Therefore the structure of the communication system must limit the effect changes in one module have on the other modules. The modules will have clear distinct responsibilities and provide an clear interface to hide the inner details of the implementation.
This also allows the system to be used on different platforms since the modules can be changed or switched without affecting the rest of the web application. Furthermore the
25
application will be designed in a manner that makes swapping compression codecs and transport protocols possible without changes to the application.
Documentation
To make the code more readable comments in the code shall be included and system documents will be provided.
7.1.2. Overview
An efficient flexible design is needed for the transport of different types of (visual) data over Internet. The transport time and quality of the data are the two main concerns.
The main focus of attention depends on the type of the data that is to be transmitted. The maintainability of the system and the support that the system offers to the user are also a concern. Reliability is less important because the system will be part of a prototype that will not be used for a long extensive period.
7.2. Proposed software architecture
This paragraph contains the design for the transport system of data from the streaming server to the client.
7.2.1. Overview
The system must uphold the requirements for different types of data in varying situations, to provide a flexible data transport. Flexibility has been achieved by separating the actual data transport from the management of the data transport. This allows the system to change the configuration of the data transport during run-time in reaction to changes at the Internet.
26
7.2.2. Subsystem decomposition
Fig 7.1: The modules of the system
Both client and server have a Transport module that handles the transport of data from server to client. At the server the module is responsible for the transmission of data, at the client the module is responsible for the reception of the data. At the client a second module, the DataManagement module, is used. This module is responsible for the storage of persistent data for later use.
The Transport module has been split further into the three submodules,
TransportManager, CompressionModule and ProtocolModule.
TransportManager
The TransportManager is responsible for the management of the data transport. The
TransportManager has two tasks.
The first is to represent the Transport module to other modules. Requests and information concerning data transport are handled by the TransportManager.
The second task is to realize the requirements that the data imposes on the transport.
The module monitors periodically the data flow and decides whether a change of strategy is required. If so the manager swaps the codec and/or protocol or changes the settings of the data transport.
The TransportManager uses the other two modules for the actual transport of the data.
27
CompressionModule
The CompressionModule is responsible for the compression and decompression of data.
The module contains multiple codecs. The TransportManager selects one of them, and switches to another if necessary.
ProtocolModule
The ProtocolModule is responsible for the transport of data. Multiple protocols such as
RTP are contained in this module. The TransportManager can use the protocols directly for the transmission of data.
7.2.3. Hardware/software mapping
The used protocols and codecs will be acquired as off-the-shelf components to limit the development time.
The Transport module runs on the Tecnai PC and on the client. If possible the available hardware is used to accelerate the processing of the data. Video cards might be used for the compression and decompression of the data. The DataManagement module is used only at the client.
7.3. Subsystem services
TransportManager setRequirements(dataType:DataType, newRequirements:Requirements)
SetRequirements is used to set the requirements to newRequirements , which the module upholds during the transmission of data of the type datatype .
ProtocolModule getProtocol(ProtocolName: ProtocolName) : Protocol
This service returns the requested protocol to the caller. The desired protocol is specified by ProtocolName .
28
CompressionModule getCodec(CodecName: CodecName) : Codec
This service returns the requested codec to the caller. The desired codec can be specified with CodecName .
DataManagementModule storeData(source:Memory, destination: Folder, dataType:DataType)
This service stores the data located at source on the hard disk in Folder . DataType is used to specify the type of the data.
Session Protocol
The server and client require a protocol for communication regarding the data transmissions and their start and ending in particular. SIP (Session Initiation Protocol) [8] and RTSP (Real-Time Streaming Protocol) [9] are the most common used session protocols. Both are equally suitable for the system. The RTSP protocol will be used in this system since it belongs to the same group of protocols as RTP(Real-Time
Transmission Protocol) and RTCP(Real-Time Control Protocol). The RTP and RTCP were selected in the literature study as data transport protocol and feedback protocol respectively.
Figure 7.2 explains how the system uses the RTSP protocol.
Figure 7.2: The communication of RTSP between server and client
1:
A. Using the url address of the streaming server, which is retrieved from a web server, the client informs the streaming server that it wants data of one of the media sources by sending the RTSP request SETUP to the streaming server. This request also contains a list of the protocols that the client supports and ports that the client can listen to.
B. If the server accepts the request a reply message is send back containing a unique id of the session. The transport protocols and ports chosen by the server for this data transport are contained in the reply as well. The session id is used in later exchanges to identify this session. If the server does not or cannot accept the request NOK is replied.
29
2:
A. The client instructs the streaming server to begin transmitting the desired data by calling the command PLAY using RTSP.
B. The server sends a reply message OK and starts transmitting data.
3:
A. The client ends the transmission by calling the TEARDOWN command of the server.
B. The server replies OK and stops the transmission.
30
8. Detailed design (ODD)
This chapter contains the main parts of the Object Design Document (ODD), which describes the detailed design of server and client.
8.1. General
8.1.1. Object design trade-offs
Two designs were considered. In the first design a process is created for each data transport. A process supports only one client. The second design allows multiple clients to join an existing process for a particular data transport. This design thus reduces the total number of running processes and in effect increases memory and processor efficiency. This makes the system more scalable because more clients can be supported.
The disadvantage of the second design is that the settings of the processes have to be balanced between the requirements of different clients. The first design offers a client oriented and less complex data transport.
In most cases only about one or two clients will be present. Therefore the second design will be used. The system shall adapt to the number of clients and resource use intensity.
8.2. Packages
This paragraph discusses the packages of the application. The data transport system consists of a streaming server and a client. The packages of the server shall be described first, followed by a description of the client packages.
31
8.2.1. Server
Fig 8.1: The packages of the Server
The DataTransport Module consists of four packages that are located at the server. The
TransportManagement Package encapsulates classes that are used to manage the data transmission. The Codec Package encapsulates the codecs that are used for compression.
The Protocol Package contains the protocols that are used to send data to clients. A global package, the StreamingServer Package, envelops all the classes and packages of the server.
32
8.2.1.1. TransportManagement
Fig 8.2: The classes of the TransportManagement package.
The TransportManagement Package consists of the classes RTSPServer, RTCPServer,
RTCPPacket, ClientManager, DataStreamList, DataStream, TaskScheduler, DataSource and Requirement. This package contains the main components of the streaming server that handle the data transport processes and communicate with the client.
Datastream
Fig 8.3: The relations of the DataStream class
The DataStream class represents the transmission of data from one source to multiple clients. Clients represented by the ClientManager class, can share a DataStream object to reduce the burden on the memory and processor. Clients may join, leave and create a
DataStream object.
The data is compressed with a Codec object and then transmitted to all the clients.
DataStream contains one protocol for multicast transmission. Clients can either use this
33
protocol or provide their own unicast protocol to the DataStream object. The Codec used by DataStream is assigned to it at the creation and cannot be changed.
The class DataSource represents a source of data on the computer. This class is passed as an parameter to point to the location of the data in memory of the computer.
ClientManager
Fig 8.4: The relations of the ClientManager class
The ClientManager class represents the client and is responsible for the management of the connection with the client. A high framerate and visual quality is maintained for each datatransmission. The available throughput is distributed according to need over the different data transmissions.
The class connects to an appropriate DataStream for each data transmission. This selection is based on the codec and compression ratio that each DataStream uses. A different compression ratio may be requested if necessary. The ClientManager uses
DataStreamList to create, join and leave datastreams. Periodically the status of the connection and client is received in RTCP packets through the RTCPServer class.
The class Requirement represents the criteria for visual data. A Requirement object is created for each type of data (RT, FCCD, SCCD) that is used. ClientManager and
DataStream use Requirement to manage the data transmission. A Requirement object is contained in ClientManager for each type of data. So ClientManager is connected to three
Requirement classes.
DataStreamList
Fig 8.5: The relations of the DataStreamList class
The DataStreamList class contains all the DataStream objects. When a ClientManager wants to create, join or leave a DataStream this is handled by the DataStreamList class.
34
TaskScheduler
Fig 8.6: The relations of the TaskScheduler class
The TaskScheduler determines in which order the DataStreams will run. The
TaskScheduler also allows ClientManager objects to communicate periodically with the clients.
RTCPServer
Fig 8.7: The relations of the RTCPServer class
The RTCPServer is responsible for the exchange of RTCP packets with a client.
Incoming RTCP packets of the client are passed on to the ClientManager for handling.
All the RTPProtocols that transmit data are known by the RTCPServer.
The class RTCPPacket represents an RTCPPacket. This class is used to pass on incoming RTCPPacket to the ClientManager.
RTSPServer
Fig 8.8: The relations of the RTSPServer class
The RTSPServer class represents the Streaming server to the clients using the RTSP protocol. This class is used to communicate with the clients regarding the start and ending of data transmissions. RTSPServer supports the RTSP requests SETUP, PLAY and
TEARDOWN.
35
8.2.1.2. Codec Package
Fig 8.9: The classes of the Codec Package.
The classes in this package handle the data compression. Classes in the
TransportManagement Package use this package to compress data prior to the transmission.
The abstract class Codec represents all the codec classes in the package as these classes are inherited from Codec. Data is accessed in memory and after processing stored in a temporary buffer.
The class CodecGenerator enlists all the codecs that the system supports. Only one instance will be active per process. The ClientManager uses CodecGenerator to create new codec objects.
8.2.1.3. Protocol Package
Fig 8.10: The classes of the Protocol package.
The classes in this package handle the transport of data to the client.
The abstract class Protocol represents all the protocol classes in the package as these classes are inherited from Protocol. The data is accessed in a temporary buffer and sent to a client.
36
The class RTPProtocol handles transport of data using the RTP protocol. This class registers status information during transmission which the RTCPServer can use to form
RTCPPackets.
The class ProtocolGenerator enlists all the protocols that the system supports. Only one instance will be active per process. The ClientManager uses ProtocolGenerator to create new protocols.
The class NetworkAddress represents the address of a computer on a network or on the
Internet. This class contains an IP address and port. The protocol classes use this class as a destination for the data transmission.
8.2.1.4. StreamingServer Package
Fig 8.11: The classes and packages of the StreamingServer package
This package contains the classes StreamingServer and Data.
StreamingServer
Fig 8.12: The relations of the StreamingServer class
StreamingServer represents the TransportModule on the Tecnai PC. All requests of other applications on the Tecnai PC directed to the TransportModule are handled by this class.
37
RTSP server is created by the StreamingServer class at the initialization of the class to handle all the RTSP requests from clients. TaskScheduler is also created by
StreamingServer during the initialization.
The Data class represents raw data (series of bits) stored in memory.
8.2.2. Client
Fig 8.13: The classes and packages of the client.
8.2.2.1 TransportManagement
Fig 8.14: The classes of the TransportManagement package.
The TransportManagement Package contains the classes StreamingClient,
ServerManager, DataStream, RTCPClient, RTCPPacket and DataDestination. The
38
ServerManager class contains the functionality of the ClientManager and RTSPServer class. The DataDestination class is the equivalent of DataSource.
At the server multiple Clients can join a single DataStream. However at the client only one Server per DataStream is possible. Thus there is no need for a DataStreamList class.
Servers create a DataStream directly when required. To make the swapping of codecs less time consuming DataStream objects at the client automatically change the codec when required.
Decisions regarding the compression and transfer of data are made at the server. The client only needs to react to decisions made at the server and to communicate with the server. The ServerManager class is therefore less complex. ServerManager is responsible for the high level communication with the server and takes care of the creation and ending of DataStreams for a particular Server. The ServerManager also handles the RTSP protocol interaction with the server.
Because the client mainly responds to the server no TaskScheduler is needed. Data is processed when it arrives. The ServerManager contains a timer which activates the
ServerManager periodically for communication with the server. The other classes in this package are identical to the classes of the TransportManagement package at the server.
The Requirement class is not required on the client and was therefore not included.
8.2.2.2 Codec Package
Fig 8.15: The classes of the Codec package.
The classes of this package are the same as the classes on the server. The only difference is that the codec classes decompress data instead of compressing.
39
8.2.2.3. Protocol Package
Fig 8.16: The classes of the Protocol package.
The classes of this package are the same as the Protocol classes on the server. The only difference is that the protocol classes receive data and store it in memory instead of transmitting data.
8.2.2.4. StreamingClient Package
Fig 8.17: The packages of the StreamingClient Package.
This package contains the StreamingClient class. StreamingClient represents the
DataTransport Module on the client, it handles all the requests concerning data reception.
A ServerManager object is created by StreamingClient for each new data server.
40
8.3. Classes
This section lists the server classes.
8.3.1. DataStream
Fig 8.18: The DataStream class
The DataStream class is part of the TransportManagement Package. A data transmission from one source to multiple clients is represented and managed by this class. DataStream objects use only one codec, assigned to it on the creation of the object. There is however one protocol for each client that is connected to the DataStream object. When the
DataStream is activated by the TaskScheduler it will send a number of frames to the clients that are connected to this datastream.
Periodically clients may request a different compression ratio for the codec or a different transfer rate for the multicast protocol. When such a change is applied all clients are informed.
The method getID returns the unique identifier of this DataStream object.
The method processData activates the DataStream object. The object responds by processing and transmitting the next nrOfFrames frames of data.
The method changeRatio can be called to request a different compression ratio for the codec.
The method getRatio returns the currently used compression ratio.
The method getCodecName returns the type of codec used as a string.
41
The method getMultiProtocol returns the name of the multicast protocol used by this
DataStream object.
The method getProtocol returns the unicast protocol used by the ClientManager object.
The method getDataName returns the name of the data that is handled by this
DataStream object.
The method addClient adds a client to the DataStream object. After a call to this method data is also transmitted to the particular client using the unicast protocol protocol .
This method is overloaded to allow the client to join a multicast protocol.
The method removeClient removes a client from the DataStream object. After a call to this method DataStream no longer transmits data to the particular client.
The method getClientCount returns the number of clients that are connected to the
DataStream object.
The method getClents returns the client that are connected to this object in a vector.
The method DataStream creates a new DataStream object.
The method ~DataStream terminates the DataStream object.
8.3.2. DataStreamList
Fig 8.19: The DataStreamList class
The DataStreamList class is part of the TransportManagement Package. All DataStream objects are contained in the DataStreamList class. Clients can join, create and leave dataStream objects through this class.
The method createStream creates a new datastream. The new object is returned to the caller. This method is overloaded to allow the ClientManager to either pass on a unicast protocol or to join the multicast protocol of the DataStream object.
The method joinStream can be called by a client to connect to an existing DataStream.
The argument protocol is the unicast protocol that the DataStream object should use to transmit data to the client. This method is overloaded to allow the ClientManager to either pass on a unicast protocol or to join the multicast protocol of the DataStream object.
The method leaveStream can be called by a client to disconnect from a DataStream.
When there are no other clients connected to the DataStream it is terminated by
DataStreamList to free processor and memory usage.
42
The method enumStreams returns a list of the DataStreams objects. The returned list contains the objects that manage the transmission for a specific data source using a particular codec type.
The method DataStreamList creates the DataStreamList object.
The method ~DataStreamList ends the DataStreamList object.
8.3.3. ClientManager
Fig 8.20: The ClientManager class
The ClientManager class is part of the TransportManagement Package. ClientManager is responsible for the management of the connection with a client. Client decides for each data transmission which DataStream should be used.
Periodically the process method of ClientManager is called by the TaskScheduler which enables the ClientManager to send an RTCP packet to the client. If a RTCP packet sent by the client is received by the RTCPServer the method handleRTCP of the
ClientManager is called which starts a reevaluation process. During this reevaluation the client decides for all the data transmissions whether the compression ratio and/or codec should be changed.
The method startTransmission starts a transmission of data from a source to the client. A client object responds to a call to this method by either joining or creating a DataStream.
The parameter sessionID is the id of the corresponding session used by the RTSP server.
Also the protocols and ports supported by the client are passed on as arguments.
The method stopTransmission ends the transmision of data of a data source to the client.
The method notifyOfChange informs the ClientManager of a change of the settings of a particular datastream.
The method process can be called by the TaskScheduler to allow the ClientManager to communicate with the client.
43
The method handleSet can be called by the RTSPServer to let the ClientManager handle the SET request of the client.
The method handleAnnounce can be called by the RTSPServer to let the ClientManager handle the ANNOUNCE request of the client.
The method handleRTCP informs the ClientManager of an incoming RTCP packet.
This starts an evaluation process which may result in a change of codecs and/or compression ratios.
The method ClientManager creates a ClientManager object.
The method ~ClientManager ends the ClientManager object.
8.3.4. RTSPServer
Fig 8.21: The RTSPServer class
RTSPServer is part of the TransportManagement Package.
The RTSPServer class handles the communication in the RTSP protocol with clients.
Clients can send RTSP requests to this class.
The method announce sends the RTSP command announce to the client. The message message is send along with this command.
The method set sends the RTSP command SET to the client. The variables are passed along with the method as a list of variables. The values are also provided as a list of values. The first element of the variables corresponds to the first element of the values, the next to the next element and so on.
The method get sends the RTSP command GET to the client. The variables of which the caller desires the value is passed on as a list. The values of the variables are returned as a list.
The method handleMessage can be called to inform the RTSPServer of an incoming message.
The method RTSPServer creates an RTSPServer object. The port that RTSPServer uses is passed on as the argument port . After a call to this method the RTSPServer waits for and handles all incoming RTSP requests.
44
The method ~RTSPServer ends the RTSPServer object.
8.3.5. TaskScheduler
Fig 8.22: The TaskScheduler class
TaskScheduler is part of the TransportManagement Package.
This class activates DataStreams successively by calling the method processData , the fraction of frames that the object may process is passed along as an argument.
Periodically the TaskScheduler also allows ClientManager objects to communicate with the clients by calling the process method.
The method addStream adds a DataStream object to the TaskScheduler.
The method removeStream removes a DataStream object from the TaskScheduler
The method addClient adds a ClientManager object to the TaskScheduler
The method removeClient removes a ClientManager object from the TaskScheduler.
The method scheduleNext instructs the TaskScheduler to activate the next object.
The method TaskScheduler creates a TaskScheduler object.
The method ~TaskScheduler ends a TaskScheduler object.
45
8.4. Sequence diagrams
8.4.1. StartTransmission
Fig 8.23: The StartTransmission sequence diagram
The client sends a SETUP request to the streaming server to initialize a datatransmission.
The request of the client also contains a list of the protocols and ports that are supported by the client. This command is received by the RTSPServer object which responds by creating an unique ID for this session and sending the sessionID back to the client along with a list of the protocols and ports that are to be used in this data transport.
When no ClientManager exists for the client a new object is created. The
TaskScheduler is informed of each new ClientManager with the call to the method addClient .
46
The client then sends the PLAY request. At reception of this request the RTSPServer starts the transmission by calling the startTransmission method of the corresponding
ClientManager object. In this method the protocols that the client supports are passed on as a list of strings. The ClientManager makes a selection of these protocols and decides which codec is the most suitable for this particular transmission. Then the ClientManager requests which DataStreams are available for a particular data source and codec to the
DataStreamList ( enumStreams ).
The settings of the returned datastream objects are investigated by the ClientManager.
This inquiry may lead to the decision that none of the existing data streams are appropriate so that a new DataStream object has to be created with a call to the method createStream of the DataStreamList object.
In this case however the ClientManager decides to join one of the existing DataStream objects. A Protocol object is created by calling the createProtocol method of
ProtocolGenerator. This Protocol object is passed on as an argument with the method joinStream that is used to join the DataStream object. The DataStreamList informs the
DataStream object by calling the method addClient . The new Protocol is added to the
RTCPServer when the chosen protocol is RTP so that RTCPServer knows all the used
RTP protocol objects.
47
9. Implementation
This chapter discusses the implementation of the server and client. A number of libraries were used, so an explanation of the choice of libraries and tools shall be given first. Then the code shall be described on a global level. Problems encountered during the implementation slowed down the progress. These problems are discussed in section
‘implementation problems’.
9.1. Libraries and tools
Programming language
The programming language C++ was selected because of its efficiency, flexibility and widespread use. The object oriented nature of C++ also makes it easy to implement well structured and modifiable code.
Codecs
Table 9.1 Investigated codecs
Investigated codecs
Name
PNG
BTPC
SPIHT
IJG
JASPER
Description Suitability
Lossless codec for greyscale images [10] . Suitable for compression of slowscan CCD because
BTPC is a general-purpose image coding scheme suitable for compression of all kinds of image [11] .
A new commercial wavelet based efficient image compression codec. Won an award
[12] it does not distort the image.
Encoding is to slow. Does have a lot of potential.
Is promising, but when tried out on microscopic images gives an abnormal
An open source library of JPEG created by the independent JPEG group [13] distorted image.
Suitable, efficient and highly portable.
An open source implementation of JPEG in Java [14]
Not suitable. Is portable, but lacks the necessary efficiency.
KAKADU A commercial codec for JPEG2000 [15] Suitable. Offers efficiency, portability and a good signal to noise ratio.
Not suitable CALIC A new context based adaptive lossless image codec [16]
HCOMPRESS A wavelet lossy codec that can be used on Not suitable: Astronomical
48
COMPFITS
16 bit images. Was designed for compression of astronomical images
Software for compression of astronomical images [18]
[17] pictures have properties that are to different from microscope pictures.
Might be suitable for diffraction images.
Not suitable: Astronomical pictures have properties that are to different from microscope pictures.
Might be suitable for diffraction images.
MPEG
H.26X
Codecs based on MPEG such as Dali,
Mpeg2Tool, Mpeg2v12 and mpeg_encode-
2.5b-src [19]
Codecs based on a H.26x format.
H261CAP/PLAY and H263CAP/PLAY
[20][21][22]
Might be suitable. Have not yet been tested as kakadu and ijg already gave the desired performance. It would have taken to much time to test these codecs as well.
Table 9.1 enlists the libraries that were considered. Implementations of formats discussed in the literature study were investigated, but other formats such as SPIHT have been evaluated too. Most of the codecs do not meet the efficiency and portability criteria.
Some are specialised (such as COMPFITS) for pictures with different properties from microscope pictures. Others are too slow such as Jasper, or do not provide enough portability or both. In table 9.1 the reason for (not) choosing these codecs is given. Of the codecs the open source IJG and commercial Kakadu are the most promising, both providing portability and efficiency.
Kakadu and IJG were compared on visual quality and compression time. As described in chapter 5, “Research”, Kakadu offers acceptable visual quality at 0.1 bpp and IJG at
0.3 bpp. It was somewhat problematic to test Kakadu because a license was required for direct access to the library. There are however a number of demo programs of this library that have been used to measure the compression time.
Table 9.2 shows the encoding time of IJG and Kakadu on an AMD Athlon 64 XP with 1
Gigabyte memory and Windows XP. General lists the time required to transport an
FCCD picture at different compression ratios. IJG and Kakadu show the time required to compress at that ratio and the total time, which is compression, transport and decompression. Decompression was assumed to take as long as compression as no accurate information could be collected on that aspect.
When IJG is set at .4 bpp and Kakadu at .2 bpp the former provides an overall faster transmission for a bandwidth of 0.5+ MB/s when compression, transport and decompression time are all taken into account. The IJG library is therefore used to compress FCCD and RT data.
Table 9.2: Shows the performance of IJG and Kakadu.
49
Ratio
( bpp)
General
Data
Size(kB)
Transmission
Time(ms)
IJG/Kakadu performance
Encoding time(ms)
IJG
Total
Time
(ms)
Frames/ sec
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
6,5
9,7
12,9
16
19,2
22,3
25,7
28,8
31,9
6,35
9,47
12,60
15,63
18,75
21,78
25,10
28,13
31,15
6,41 19,17
6,71 22,89
7,19 26,98
7,5 30,63
7,97 34,69
7,81 37,40
7,97 41,04
8,91 45,95
9,22 49,59
52
43
37
32
28
26
24
21
20
Encoding time(ms)
Kakadu
Total
Time
(s)
Frames/ sec
15,75 37,85
19,5 48,47
23,5 59,60
23,5 62,63
23,5 65,75
27,5 76,78
27,5 80,10
27,5 83,13
31,25 93,65
26
20
16
15
15
13
12
12
10
Graph 9.1: Encoding time by Kakadu and IJG for different compression ratios.
1
0,9
0,8
0,7
0,6
0,5
0,4
0,3
0,2
Kakadu
IJG
0 0,005 0,01 0,015 0,02 0,025 0,03 0,035
Encodingtime (sec)
Graph 9.2: The total time, (compression, transport and decompression) required by Kakadu and
IJG
50
1
0,9
0,8
0,7
0,6
0,5
0,4
0,3
0,2
Kakadu
IJG
0 0,02 0,04 0,06
Total time (sec)
0,08 0,1
RTP libraries
The RTP protocol can be implemented with an existing RTP library, a data transport tool that supports UDP or with sockets. Most toolkits and sockets use the Winsock of C++, which is based on the Windows platform. These tools and sockets are not portable enough, though other platforms do partially support Winsock. One tool, the ACE
(Adaptive Communication Environment) toolkit , was however specifically designed to be portable and efficient [23]
The only suitable C++ RTP library is JRTPLIB [25] . The Java Media Framework (JMF) can also be used, but is based on Java and is not efficient enough [26] . Aside from these libraries a RTP/RTCP implementation is provided by the open source community Vovida
[24] . Because JRTPLIB was discovered only late in the implementation phase the Vovida classes have been used for the handling of the RTP/RTCP protocol. With the Vovida classes more direct control on the RTP and RTCP protocol and data transfer can be maintained without building the code from scratch. The actual data transport is handled with the ACE toolkit.
9.2. Code structure
The ‘pipe filter’ paradigm is used to represent the data flow from server to client. The
ACE toolkit contains two classes that correspond to this paradigm, the ACE_Stream class
(pipe) and ACE_Task class (filter). A number of classes are based on this stream class and represent the pipe in the paradigm. Other classes are based on the Task class. There are also a number of support classes that were created to be used in a stream and/or task
51
classes. A global discussion of the stream classes will be given first followed by a discussion of the task classes. Only the main classes will be treated, for a detailed description of all classes see Appendix A.
General
Stream
Fig 9.1: The stream classes
To hide the complexity of ACE_Stream the abstract class Protocol_Stream was created which forms a wrapper around the class. Protocol_Stream represents a stream in which data flows through successive filters (or protocol layers). Reception of data will be handled automatically, as explained later, so Protocol_Stream only offers the method send for transmission of data. Clients can be added and removed to a stream, a
Protocol_Stream knows all the clients to whom the data has to be sent.
The class is abstract, it is not able to perform any operation on the data. The processing of data has to be done in specialised classes that inherit from Protocol_Stream. In this implementation two such classes were created, Rtp_Stream and Rtcp_Stream.
Rtp_Stream handles the transmission and reception of the visual data with the RTP protocol. See figure 9.2 for a global view of the steps (filters) involved in the transport of data from server to client in the RtpStream. The blocks in the picture represent the classes that are used.
Fig 9.2: The dataflow in RtpStream
52
Rtp_Stream can only be used for either transmission or reception of data but not both. In case of transmission the data is retrieved from memory, compressed, packed in one or more RTP packets and transmitted to all clients. For reception this sequence is reversed.
The type of codec to be used for compression and the buffer for the data storage are assigned to the Rtp_Stream object at creation.
Rtcp_Stream does not transmit data, but does sent feedback with statistics of the dataflow of the Rtp_Stream. The RTCP protocol is used for the transmission and reception of this feedback. The transmission in the Rtcp_Stream involves packing statistics in a RTCP packet and transmission of this packet. At the server there is one Rtcp_Stream for each client per datastream. At the client there is one Rtcp_Stream for each Rtp_Stream. The
Rtcp_Stream has to know the RTPProtocol object that is used in Rtp_Stream to be able to collect information regarding the transport.
Because Rtp_Stream and Rtcp_Stream are closely correlated the class ServerStream for the server and ClientStream for the client were developed to encapsulate these two classes. ClientStream represents a data stream at the client, ServerStream a data stream at the server. At creation of the streams a codec for (de)compression and a buffer for data storage are assigned to the streams
Task
Fig 9.3: The Task classes
The Stream classes do not process the data themselves. That is done in the Task classes that inherit from ACE_Task. Multiple tasks can be added in sequence with an
ACE_Stream object. When data flows through the stream each task processes the data and passes it on to the next task in the stream in the ACE class ACE_Message_Block.
The direction can be upstream (transmission) and downstream(reception).
53
The abstract class Protocol_Task inherits from the ACE class ACE_Task. Protocol_Task represents a single task that processes the data according to some protocol. With the method put (or put_next) the data can be passed on to the next Protocol_Task object.
Classes that are used for processing of data in a stream all inherit from Protocol_Task.
Upstream (transmission) data processing is handled in the method send(ACE_Message_Block *message) downstream(reception) in the method recv(ACE_Message_Block *message) . In Protocol_Task these methods are abstract, specialised classes need to implement either send or recv or both methods.
The different operations that are performed on the data in RtpStream, (see picture 9.1), memory retrieval/storage, compression/decompression, packing/unpacking and transmission/reception are taken care of in specialised classes that inherit from
Protocol_Task. These classes are respectively DataHandler, Codec, RtpReceiver,
RtpTransmitter and TransportProtocol.
In Rtcp_Stream the classes RtcpReceiver, RtcpTransmitter and TransportProtocol are used.
Reception of data is handled passively; once data arrives it will be retrieved from the network/Internet. This method solves a problem that was encountered during the implementation.
In the first approach the objects responsible for reception of data waited actively for incoming data. The protocol RTP uses a ‘best effort’ approach, packets are only send once, so the reception objects can miss some of the sent packets which results in lost packets. This poses no problems when only one stream is active. But once more streams were activated the number of packets lost increased considerably, which reduced the performance. The reason for the increase in lost packets is that only one object can listen for data in a single thread approach. In case of for instance two streams half of the incoming data will be intended for the inactive object. This results in a drop of frame rate.
This might be prevented by creating a process or thread for each receiver. But an even worse performance was achieved with this approach even though all packets did arrive at their destination. This result is probably due to the extra overhead required to maintain the separate threads/processes.
The chosen solution does not lose packets (except for transmission loss on network/Internet) and is relative simple.
9.3. Implementation problems
During the implementation a number of problems were encountered that slowed down the progress of this phase.
Due to the unfamiliarity with the IJG library it was unclear how the library had to be compressed and used. Once the library was successfully applied for (de)compression of
54
pictures the memory and error handling of the library had to be replaced. During compression and decompression IJG stores temporary data (pictures) on disk which slows down the process. Errors produced by corrupt JPEG pictures were printed to screen and could lead to a shut down of the system. Again however the unfamiliarity with the tool made it problematic to make the necessary changes.
The Vovida classes contained calls to methods that were not supported by the used C++ version. The compilation of these classes therefore produced errors. By adapting these classes those errors were removed, but the time required for these modifications reduced the gain in development time that I hoped to achieve with using the Vovida classes.
The ACE toolkit did not give these kinds of problems, but due to the steep learning curve of the tool and the lack of clear documentation it did take a long time to learn to work with this toolkit.
Most of the other problems encountered during the implementation were memory problems such as incorrect data, memory leakage and attempts to write to unreserved memory. The causes of these problems were difficult to locate so solving them was a time consuming process.
After implementing the server and client the testing phase began. The client had one
Linux pc available in Eindhoven which could be used for the tests that were run over
Internet. So the server, client and libraries had to be compiled for the Linux platform.
Compilation of the ACE toolkit posed difficulties however. To make the ACE code more clear each long line of code is split over multiple lines with the marker ‘/’. The old compiler that was used did not recognize this and therefore failed to compile the code.
Installation of a newer compiler failed as well as using a different computer with Linux.
Finally, due to the time it took to obtain a successful compilation, no further attempts were made. It should be noted though that ACE is portable in Linux and that the main reason for the failed attempt was that the available Linux C++ compiler is to old.
55
10. Testing
The server and client were tested on performance and stability. The stability tests focused on quality of pictures received at the client, correct order of arrival of the pictures, and stability of the server and client. The performance tests measured the framerate (fps) and number of lost packets for different set-ups on a single computer, a network and the
Internet. Table 10.1 shows the network and Internet speeds of the used computers and table 10.2 lists their configuration.
Table 10.1 Network/Internet speeds of used computers
Network & Internet speeds
# Uploadspeed
(KB/s)
1 19.5
Downloadspeed
(KB/s)
60.3
Networkspeed
(MB/s)
12.5
2 19.5
3 12800
4 Depends on location
60.3
12800
Depends on location
12.5
12.5
12.5
5 12800 12800 12.5
Table 10.2: Configurations of used computers
Computer specifications
# Processor Memory
1 AMD Athlon 3400 64 1 GB
2 Intel Pentium 1.6 GHz 512 MB
3 Intel Pentium 2.8 GHz 512 MB
4 Intel Pentium 1,6 Ghz 512 MB
(notebook)
5 Intel 3.06 Ghz 2 GB
Operating System
Windows XP
Windows 98
Windows XP
Windows XP
Windows Server
2003
10.1. Performance tests
All performance experiments are carried out using one or two datastreams. In each test
15000 fastscan CCD (FCCD) pictures are sent to the client(s). These tests measure the framerate in frames per second (fps) and packet loss per stream. In the local computer tests the delay of each frame is measured as well. Because different computers have different clocks the delay could not be recorded in the network and internet tests.
Local computer
56
Set-up
The client and server are run on a single computer. In the local tests the data arrives directly at the client. The frame rate is therefore an indication of the efficiency of the client and server. The results obtained in this experiment can also be used for comparison with the frame rate in later tests.
The amount of fragmentation of pictures and the compression ratio are varied in the tests.
Table 10.3 enlists the specific test set-ups. The amount of fragmentation is shown in number of packets used per picture. The compression ratio is represented in bits per pixel
(bpp).
Table 10.3 Local test set-ups
#
1
2
3
4
5
6
7
8
Server &
Client
PC 1
PC 1
PC 1
PC 1
PC 1
PC 1
PC 1
PC 1
Local test set-ups
1
1
1
1
2
1
1
#Streams #Packets per picture
1 4
4
1
34
7
3
1
2
Compression ratio (bpp)
0.5
0.5
0.5
4
4
0.2
0.2
0.1
9
10
PC 3
PC 3
1
2
4
4
0.5
0.5
11 PC 3 2 34 4
Result
The test results are presented in table 10.4. At a normal compression ratio (0.03bpp-0.05 bpp) and fragmentation (4 packets per picture) the average framerate is about 40 fps for one data stream and about 19fps for two streams. The highest measured variation in frame rate (jitter) for these settings was 3.37 fps. There was only minimal difference in performance between tests on PC1 and PC3
The amount of data fragmentation had little influence on the framerate. For example test 4 and 5, that use respectively 4 and 34 packets per picture, have a difference in framerate of only 1fps. This might be due to the low loss rate of packages. When data is send over the Internet on a more restricted bandwidth ( < 400 KB/s) the fragmentation of pictures will have more influence on the framerate because of the increased chance of loss of picture parts.
Different settings of the compression ratio did however have effect on the frame rate, from 18 fps at a low compression ratio (0.5bpp) up to 48 fps with a lot of compression
(0.02 bpp). There is no data transport so the drop in framerate is caused by the increase of processing time and memory storage that is required for larger data . So highly
57
compressed pictures give better performance on computer systems than less compressed pictures even when there is no data transport over a network or the Internet.
Table 10.4 Local test results
Local test results
Test
#
Stream Average
Framerate
(fps)
Variation
Framerate
(fps)
Average
Delay
(ms)
Average
Packet arrival rate %
98,30 1 1
2
3
1
2
1
40,24
19,76
19,67
39,64
17,52
3,31
0,85
0,98
4,16
0,8
28,51
53,74
53,76
17,88
68,59
99,84
99,78
100
4
5
6
7
8
9
10
11
1
1
1
1
1
1
1
2
1
2
18,57
43
46,10
47,63
38,94
19,24
19,18
7,78
7,80
1,06
3,28
10,55
6,16
3,37
1,78
1,90
0,86
0,80
60,5
25,18
15,23
21,19
30,06
58,36
58,30
141,21
140,71
99,41
99
97,41
97,39
98,35
99,93
98,08
98,07
97,17
97,14
Network
Set-up
In the three network tests the clients are connected to the server by a 100 Mb/s network.
The compression ratio is set at 0.05 bpp and the pictures are fragmented over 5 packets.
In the third test a second client is run simultaneously.
58
Table 10.5 Network test set-ups
Network test set-ups
Test # # Streams Server Client(s)
1 1 PC1 PC4 (laptop)
2 2 PC1 PC4 (laptop)
3 2 PC1
Results
Table 10.6 Network test-results
PC4 (laptop), PC2
1
2
PC4
PC4
1
1
2
Network test results
Test # Client Stream Average
Framerate
(fps)
39,99
26,25
26,22
21,12
Variation
Framerate
(fps)
6,87
1,11
1,17
1,71
Average
Packet arrival rate %
96,53
99
99
96,23 3 PC4 1
2
PC2 1
2
16,55
11,11
14,39
3,49
3,97
3,83
94
84,13
88
The test results are similar to those of the local tests. About 40 fps for one datastream and
26 fps for two streams was measured. The average packet loss is low so the network bandwidth is sufficient and does not limit the transfer rate of packets.
When two clients are run the framerates of the streams on PC4 are lower than those measured in test 2 with a single client and on PC2 11.11 fps and 14.39 fps was observed.
The packet loss is also higher than that of test 2.
The lower framerate in test3 is a direct result of the increased packet loss. At an arrival rate of 88% of the packets (which was observed for stream2 on PC2) the chance of arrival of all five packets for a picture is 0.88
5
≈ 0.53. When 27 frames are sent per second about
27 *0.53 = 14.31 frames of these 27 frames will arrive intact; 14.39 fps was actually observed.
59
The packet loss might be due insufficient network capacity but the lack of processing power and/or memory of PC2 can also lead to this result. The different OS Windows 98 instead of Windows XP might be another cause.
Internet
Set-up
In the following tests the server sends data to the client over Internet. The fragmentation is set at 5 packets per picture and the compression ratio at 0.05 bpp. In the fifth test the effect of a low compression ratio of 0.5 bpp is measured. Then the compression ratio is reduced and the fragmentation increased. The server and client have an Internet connection of 100Mb/s. No tests were carried out on lower bandwidths because there was no suitable computer available with such a bandwidth.
Table 10.7 Internet test set-ups
4
5
6
7
Internet test set-ups
Test # Server Client #Streams #Packets per
1
2
3
PC3
PC3
PC5
PC5
PC5
PC3
1
2
1 picture
4
4
4
PC5 PC3
PC5 PC3
PC3 PC5
PC3 PC5
2
2
2
2
4
34
4
6
Compression ratio (bpp)
0.5
0.5
0.5
0.5
4
0.5
0.6
8
9
10
11
PC3
PC3
PC3
PC3
PC5
PC5
PC5
PC5
2
2
2
2
8
15
16
17
0.8
1.12
1.12
1.12
12
Results
PC3 PC5 2 18 1.12
The results of the tests were again similar to those of the network and local tests. There is however more jitter, see graph 10.1, and a higher packet loss in tests 1 and 2. Despite this the framerate is still about 18 fps for both streams in test 2 and 32 fps for test 1.
The performance in test 5 was poor but similar to the results of the local test 11. So this result seems to be due to computer configuration rather than the available bandwidth.
60
The framerate at lower compression ratios and higher fragmentation settings is slightly lower than that at higher settings. The difference is however minimal and even at 1.12 bpp with a fragmentation of 18 packets per picture the system still performed satisfactory.
Graph 10.1: The framerate during test 1
Performance
70
60
50
40
30
20
10
0
#Received frames
Table 10.8 Internet test results
Internet test results
Test # Stream Average
Framerate
(fps)
Variation
Framerate
(fps)
1
2
1
1
7,62
4,25
Average
Packet arrival rate %
88,83
88,03
3
4
5
6
7
2
1
1
1
2
1
2
1
2
32,66
17,81
18,06
39,36
20,11
20,10
7,81
7,79
26,36
26,46
26,28
4,25
2,68
0,80
0,79
0,80
0,83
3,30
3,55
3,46
88,24
99,76
98,41
98,24
98,16
97,96
92,42
92,22
92,49
61
2
1
2
1
2
1
2
1
2
1
2
8
9
10
11
12
26,38
25,25
25,08
25,17
25,16
22,78
22,74
24,24
24,22
22,34
22,07
3,45
3,32
3,25
3,33
3,38
2,89
3,01
2,16
2,11
3,60
4,26
92,33
91,44
91,40
91,86
91,42
93,34
93,23
93,80
93,69
92,68
92,50
10.2. Operational tests
In the first two operational tests the server provides two simultaneous datatransmissions to two clients. One of the clients is located on the same computer as the server, the other is located on a second computer connected to the server through the Internet. In the last test the server and client are run on the same computer.
Data correctness test
Set-up
Different FCCD and SCCD pictures are send to the client, total 20 pictures are transferred. Also an empty picture is transmitted to test the behaviour of the system with such data. One picture will be altered before decompression to simulate an error burst and should be discarded by the system. The pixel values of received pictures will be compared in RMSE with the originals.
Result
None of the received pictures had unusual visual effects. The average RMSE is quite low,
38, with minimal variation. So the pictures were reproduced correctly at the client. The empty and corrupted pictures were also processed without any problems.
62
Arrival order test
Set-up
To test whether the pictures arrive in the correct order a sequence number is introduced in the frames which is increased for each successive picture. The client verifies whether the sequence number of received frames does increase.
Result
A total of 15000 frames were transmitted by the server, no incorrect order of frames was detected.
Stability test
Set-up
The client and server are run for 3 hours on two different computers. Then the effects of a crash and restart of the server are simulated by closing and starting the server. Also a client is shut down and restarted while the server was running. After transmission of
100000 frames one data stream is ended by the server.
Result
The client and server continued to operate correctly during the 3 hour period.
10.3. Testing Problems
The local and network tests were successfully performed, but the Internet tests have been more problematic. In the first attempts with data transport over Internet the data could not be reassembled or was not received at all at the client. Locating and solving these problems was difficult for two reasons: a lack of suitable testing environment and the dependency on people from outside of the project.
Lack of suitable testing environment
No suitable business computers were available to run the client on so privately owned computers were used instead. These computers have Internet connections through firewalls and/or routers. Firewalls can block or slow down incoming data. Routers have to be configured correctly before they deliver data to the appropriate computer. Some of the computers are old systems with low memory, processing power and/or bandwidth.
And some have Windows 98 as Operating System while the server and client have been compiled for Windows XP. All of these factors can lead to data reception problems.
63
An ideal set-up therefore should have a powerful PC for the client (1 Gigabyte memory, 2
Ghz processor) and a Internet connection of at least 300 kB/s that does not go through a router. In the best case there should be no or only a minimal firewall. This project however had to cope with less than ideal set-ups which made it more difficult to locate and solve any data reception problems.
Dependence on people from outside of the project
This project is carried out by only one person so someone from outside of the project was required at either the client or the server. Due to this dependency it was harder to observe and solve the problems that were encountered. It also meant that people had be available to assist in this project.
It took several weeks before the data reception problems were solved. On some systems the firewall was blocking the incoming data. Once correctly configured the data was received as intended. On the other two computers the problem was solved by replacing the router with a modem. It also turned out that on low bandwidths (<300 KB/s), which all the privately owned computers have, most packets (more than 96%) got lost. Most of the pictures therefore could not be reassembled. By slowing down the transmission rate of packets the number of received packets improved considerably (almost 100% arrival rate).
.
64
11. Conclusion
This chapter reflects on the progress of the project, the achieved successes and encountered problems. Also mistakes that have been made are discussed. Then advice is given regarding further development of server and client.
The discussion of the successes, problems and mistakes of the project can be found under project evaluation. Advice regarding extensions and improvements of server and client are treated under “Further development of the server and client”.
11.1. Project evaluation
11.1.1. Success evaluation
Succesevaluation
The goal of this project as stated in the RAD is:
The project is successful when data from the microscope can be delivered to the UI. The system also must be fast enough to give users the impression that the microscope reacts almost directly to their actions. A gain in transport efficiency of the data is therefore a criteria. Efficiency in this case means the transport time required for a set level of quality of the data. The project shall be finished before 1 October 2005. There are no other objectives or success criteria.
As the test results in chapter 10, “Testing” have shown the server is capable of delivering
FCCD and RT video in real-time to the client on bandwidths from 12.5 MB/s. The amount of jitter is within acceptable range. So the main goal of this project transport efficiency has been achieved. The server and client are easy to modify and maintain because of well structured and documented code. There are also Analysis, design and implementation documents with a discussion of the decisions that were made and the reasons for those decisions.
This project also produced a number of analysis (see chapter 5 Research) that indicated that conventional methods for compression of the data are most suitable for this problem.
So there is still some work required on the server and client, but the main foundation has been laid and can be extended upon by FEI. Perhaps in some future graduation project.
Completed
65
The server is capable of sending data from multiple sources to multiple clients using the
UDP protocol. The basis of the RTP protocol is implemented and can fragment/reassemble the data if it is larger than the maximum allowed size on the network. The client also reorders the data, maintains the sequence and records information of the transmission and arrival.
Feedback of transport statistics to the server and client is provided with the RTCP protocol. Data is compressed prior to transmission with the IJG library which also decompresses it at the client.
The system contains a log with four available levels; quiet, whisper, talk and shout. At quiet no logging activities occur, at whisper only the main procedures are logged and nearly all procedures are logged at shout. The output of the log can be directed to the monitor and/or a file.
11.1.2. Project problems
Communication
At the beginning of the project it was unclear what the client and users specifically wanted from the system, how users operate and what aspects of the pictures are important to the users. During a meeting in the middle of the project with the client it turned out that he expected more focus on the aspect of compression. To meet this desire a research was started to investigate the effects of compression on the microscope pictures and the nature of the noise present in these pictures. The findings of the research can be found in chapter
5.
Every month a meeting with the client and every week with users was held.
However because the style of communicating of the client and myself were different, communication was not always successful. Meeting with the client was difficult as he was occupied with many other projects in Eindhoven, Japan and the USA. It turned out that the use of phone and email did not solve the communication problem. Users and client also should have been made more aware that certain aspects were unclear. And my style of communication should have been more assertive.
In the second half of the project the number of meetings was increased.
Time
There is a saying that the time that a software development project takes to complete is about the planned time multiplied by Π. This is of course a very crude method but does indicate that the delay for such a project can be significant. This graduation project was not different in that aspect. Throughout the project there were delays that caused the project to take longer than expected. To limit the effects of similar delays in future projects the reasons for those in this project are investigated.
66
Project approach
As explained in chapter 2 a two-phased approach was chosen for this project. Each phase consisted of analysis, global design, detailed design and implementation steps. The analysis and design phase were thoroughly documented.
It can be argued whether the extensive documentation was needed and practical. With less focus on thorough documentation of the system there could have been more focus on the implementation of the server. Then the client could have gotten a more direct impression of the system and given feedback for further development. The design and documentation might have been less sophisticated, but would have been sufficient for a prototype
Lack of experience
To guarantee a working efficient server and client the implementation concentrated on producing the vital parts of these systems. As discussed in chapter 9 “Implementation” the ACE toolkit, IJG libraries and adapted Vovida classes were used.
The first attempts to use the IJG library produced multiple errors. Once these had been solved other errors occurred. Especially the decompression process caused unexpected problems. One of these, an error raised by a call to jpeg_finish_decompress, has not yet been solved. Furthermore some time was required to learn how to use IJG. It should be stated though that IJG is a stable library.
The Vovida classes also produced errors. By adapting these classes those errors were removed, but the time that this required reduced the gain in development time that I hoped to achieve with using the Vovida classes.
The ACE toolkit did not give these kinds of problems, but it did take some time to learn how to work with this toolkit.
In general a lack of experience with these tools and libraries was the main cause of the problems.
Underestimation of the required time
A third reason for the failure to meet deadlines might be that the time required for the different steps was underestimated. More time should have been planned.
Consistency of design and implementation
The implemented code is not completely consistent with the design. New classes have been created and not all of the classes in the design were implemented. The reason for this inconsistency is that the basis of the system, the RTP/RTCP protocol and
(de)compression of data, required more time than expected. The design focused mostly on the management of that layer. During the design it was assumed that an appropriate codec and RTP/RTCP library would be available. No suitable RTP/RTCP library was however found in the beginning of the implementation. With more extensive research a library such as JRTPLIB would have been found.
67
Despite this inconsistency the structure of the implemented code does connect well with the design. Only small modifications will have to be made to the design. With more available time the new classes would have been included in the design. Then the entire system would have been implemented.
11.2. Further development of the server and client
11.2.1. Extensions and improvements
Not completed
Due to a lack of time the following parts were not implemented and were left to FEI to implement as was agreed upon with FEI.
No multicast transport protocol was implemented. Such a protocol can speed up the data transport if used on a multicast supported network.
The TCP protocol was not provided as an alternative to UDP. Some firewalls block all UDP traffic, so in some cases TCP should be used instead.
The RTSP server was not complete enough to be included in the final system..
No alternatives for the IJG codec, such as Kakadu were included. I recommend
PNG for compression of SCCD pictures because it is lossless and allows compression of 16 bit pictures; Kakadu and possibly SPIHT for compression of
FCCD and RT pictures. COMPFITS might be suitable for compression of diffraction images.
Most classes of the ODD design were only partially implemented
The system has not yet been connected with the microscope system. Thus it has not been tested completely.
Besides these extensions I also suggest to implement the Resource Reservation Protocol
(RSVP). With this protocol a certain Quality of Service can be requested on the Internet.
If supported this can improve the data transport considerably.
FEI can also improve the server and client by decoupling the codec format from the
RTP protocol. In the current implementation the RTP protocol assumes the use of the
JPEG format.
Furthermore during testing of the system, see chapter 11, it was discovered that the system consumes a high degree of the available bandwidth. This might prevent other processes that use the Internet from receiving and/or sending data. Thus there is a possible need for resource control.
Another aspect to look into is the logging of the processes and the error handling. Both are somewhat basic and need to be improved.
11.2.2. System problems
Most problems of the client and server such as memory leakage and crashing were corrected during the project. There is however still one known remaining problem.
68
In the class IjgReceiver that decompresses JPEG pictures with the IJG library, the call jpeg_finish_decompress raises an error claiming that two SOI (start of input) markers have been found. The method releases used memory so to prevent memory leaks has to be called. To disable the errors a marker is set to EOI (end of input) before calling the method.
69
Literature and references
http://research.edm.luc.ac.be/jori/jrtplib/jrtplib.html
(Liesenborgs, J.), [25] http://hfrp.umm.edu/People/Hao/ENEE728I/ENEE728I_CALIC.pdf
(Hao Hu), [16]
Hearn Donald, Pauline Baker M. (second edition, pp. 174-175), Computer Graphics ,
Prentice Hall: New Jersey, 1997, [7]
Tanenbaum, Andrew S. (fourth edition), Computer Networks , Prentice Hall: New Jersey,
2003, [4] http://cadcwww.dao.nrc.ca/ADASS/adass_proc/adass3/papers/veranj/veranj.html
(Véran
J.P., Wright J.R., Compression Software for Astronomical Images ), [18]
Zandbergen H.W., Development of Remote Instrumentation (Electron Microscope, Dual
Beam) on the National Centre of High Resolution Electron Microscopy ., [1] http://www.feicompany.com/about/index.aspx
, [2] http://www.unl.edu/CMRAcfem/em.htm
(Electron Microscopy), [3] http://www.sussex.ac.uk/Units/physics/iotweb/pages/ccds.html
(CCD’s and noise), [5] http://www.astro.utoronto.ca/~bridge/handouts/ccd.pdf
, [6] http://www.cs.columbia.edu/sip/ , [8] http://www.rtsp.org/ , [9] http://www.libpng.org/pub/png/ , [10] http://www.intuac.com/userport/john/btpc5/index.html
, [11] http://www.cipr.rpi.edu/research/SPIHT/ , [12] http://www.ijg.org/ , [13] http://www.ece.uvic.ca/~mdadams/jasper/ , [14] http://www.kakadusoftware.com/ , [15] http://www.stsci.edu/software/hcompress.html
, [17] http://www.mpeg.org/MPEG/index.html
, [19] http://ubvideo.com/public/h26l-white_paper.pdf
, [20] http://www-mobile.ecs.soton.ac.uk/peter/h263/h263.html
, [21] http://en.wikipedia.org/wiki/H.264
, [22] http://www.cs.wustl.edu/~schmidt/ACE.html
, [23] http://www.vovida.org/ , [24] http://java.sun.com/products/java-media/jmf/ , [26]
70
Terminology and abbreviations
Terminology
Bandwidth
Best effort
Bias noise
The maximum amount of data per second that the connection can transport. The actual bit rate may be lower. See chapter 4 paragraph 1.
Best effort stands for sending the packets once without guarentees for correct arrival of the data. See chapter 4, section 3.1
Noise produced by the inexact analog to digital conversion in the read-out of the CCD camera. See chapter 5, paragraph 2.
Bit rate
Charged Coupled
Device
Client
Number of bits that can be send per second.
Type of camera used in electron microscopes.
Codec
Compression ratio
Dark noise
Firewall
Flatfield noise
Fragmentation
The computer of the user that is connected with the electron microscope.
Coder/Decoder that allows for compressing and decompressing of data.
Ratio between the size of the original image and the compressed image.
Noise that is generated by heat. See chapter 5, paragraph 2.
Application that protects the computer or a local network by limiting the access from outside networks.
Noise generated by the unequal distribution of electrons on the camera surface.
Splitting of data over several packets.
Number of frames shown per second (fps). Frame rate
Fourier Transform Filter Filter that uses the Fourier Transform to reduce the noise
71
Gaussian filter
Internet Protocol
Jitter
Latency
Lossless/Lossy
Noise
Observer
Object oriented
Operator
Packet
Pipe-Filter paradigm
Pixel depth
Poisson noise
Port
Portability
A filter that uses neighbouring pixels to reduce the amount of noise. See chapter 5, paragraph 2.
A network and transport protocol used for exchanging data over the Internet (RFC 791).
The variation of the latency. See chapter 4, paragraph 2.
The delay between video capture by the camera of the microscope and display of the video. See chapter 2, section 2.
Lossless compression keeps the original image intact. Lossy compression removes visual information.
Random disturbance in the data.
The person who views the data coming from the microscope but is not operating the microscope.
Type of programming language.
The person who operates the microscope.
Generic term for a bundle of data, organized in a specific way for transmission. A packet consists in the data to be transmitted and certain control information, including the destination address.
Paradigm in which data flows through pipes.
The pipes can have filters that alter the data that passes through these filters.
Number of bits per pixel.
Noise that is due to the unpredictability of the number of electrons that fall on a pixel during a certain interval. The distribution of electrons is follows the Poisson distribution.
Entry point of a computer for data transport over a network.
The ability of a program to be run in various
72
Quality of Service
Real-Time environments and operating systems without making changes to the program.
The offered level of quality in terms of frame rate, video quality, latency and jitter.
Within a time limit of milliseconds.
Real-Time Control
Protocol
Real-Time Streaming
Protocol
The control protocol for RTP, RTCP is used for control and diagnostic on RTP sessions (RFC 3605).
An internet specification (RFC 2326) for streaming media.
RTSP can control multiple data delivery sessions, provide a means for choosing delivery channels such as UDP, multicast
UDP and TCP, and provide a means for choosing delivery mechanisms based upon RTP.
The Internet-standard protocol (RFC 1889) for the transport Real-Time Transport
Protocol of real-time data, including audio and video.
A protocol with which a certain quality of service can be Resource-Reservation
Protocol reserved on a network.
Root Mean Square Error A formula to measure the difference between the expected data and the observed data.
Router
RT
Server
Session Initiation
Protocol
Signal to noise ratio
Throughput
Transmission Control
Protocol
A device that routes data from a local area network (LAN) to another network connection.
Type of camera used in electron microscopes.
Computer at the microscope, which allows users to operate the microscope over the Internet.
Is an Internet standard protocol (RFC 3261) for initiating an interactive user session that involves multimedia elements such as video, voice, chat, gaming, and virtual reality.
The ratio, expressed in dB (decibels), between the normal signal output and the noise level within a signal.
The bit rate at which the data is being transported at a specific moment.
TCP is a transport layer protocol (RFC 793)
That enables two hosts to establish a connection and
73
exchange streams of data. TCP guarantees delivery of data and also guarantees that packets will be delivered in the same order in which they were sent.
Transport time Time to transport data from the sever to a client or from the client to the server.
User Datagram Protocol
A connectionless ‘best effort’ transport layer protocol (RFC
Unicast/Multicast
Webservice
X-Ray noise
768).
Unicast connections send a packet only to one receiver, multicast can send to multiple recipients.
A service of an application on a server that is accessible with standard webbased protocols.
Noise generated by high energetic electrons.
Abbreviations
ACE Adaptive Communication Environment
BPP Bits per pixel.
CCB Name of the computer that directly controls the microscope.
CCD
FEI
IJG
IP
Charged Coupled Device
Company that produces nanotechnology
Independent JPEG group.
Internet Protocol
JMF
ODD
OS
RAD
RMSE
Java Media Framework
Object Design Document
Operating System
Requirements Analysis Document
Root Mean Square Error
74
RSVP
RTCP
RTP
RTSP
QoS
SDD
SEM
SIP
TCP
TEM
UDP
UI
VLN
Resource-Reservation Protocol
Real-Time Control Protocol
Real-Time Transport Protocol
Real-Time Streaming Protocol
Quality of Service
System Design Document
Scanning Electron Microscope
Session Initiation Protocol
Transmission Control Protocol
Transmission Electron Microscope
User Datagram Protocol
User Interface
Virtueel Lab Nederland (Virtual Lab Netherlands)
75
Appendices
Appendix B – Implementation classes
76
Appendix A – Design (ODD)
A.1. Class diagrams
A.1.1. Server
A.1.1.6. RTCPServer
Fig A.1: The RTCPServer class
The RTCPServer class is part of the TransportManagement Package. RTCPServer is responsible for handling the exchange of RTCP packets with a single client.
The method sendRTCP sends a RTCP packet to the client which contains the statistics recorded by the corresponding RTPProtocol objects. After a call to this method the statistics of the RTP protocols are reset for the next transmission of a RTCP packet.
The method addProtocol adds a RTPProtocol to the RTCPServer object. The
RTCPServer transmits information of all the RTPProtocols it knows about.
The method removeProtocol removes a RTPProtocol.
The method RTCPServer creates a RTCPServer object.
The method ~RTCPServer ends the RTCPServer object.
A.1.1.7. Codec
Fig A.2: The Codec class
77
The Codec class is part of the Codec Package. The Codec class is abstract and represents all codecs. Codecs that the system uses are classes inherited from the Codec class. Such a class is in most cases a wrapper of an off the shelf codec.
The c ompress method compression a number of frames of the data.
The setRatio method sets the ratio at which data is compressed.
The Codec method creates a Codec object.
The ~Codec method terminates a Codec object.
A.1.1.8. Protocol
Fig A.3: The Protocol class
The Protocol class is part of the Protocol Package. The Protocol class is abstract and represents all protocols. Protocols that the system uses are classes inherited from the
Protocol class.
The transmit method transmits data to the client.
The addReceiver method adds a recipient to the Protocol object. After a call to this method the Protocol also transmits data to address .
The removeReceiver method removes the recipient specified by address .
The Protocol method creates a Protocol object.
The ~Protocol method terminates the Protocol object.
A.1.1.9. CodecGenerator
Fig A.4: The CodecGenerator class
78
The CodecGenerator class is part of the Codec Package. CodecGenerator is responsible for the creation of new codec objects.
The method enumCodecs returns a list of the supported codecs.
The method createCodecs creates a new Codec object. The new codec is returned to the caller.
The method CodecGenerator creates a CodecGenerator object.
The method ~CodecGenerator terminates the CodecGenerator object.
A.1.1.10. ProtocolGenerator
Fig A.5: The ProtocolGenerator class
The ProtocolGenerator class is part of the Codec Package. ProtocolGenerator is responsible for the creation of new codec objects.
The method enumProtocols returns a list of the supported protocols.
The method createProtocol creates a new Protocol object. The new Protocol is returned to the caller.
The method ProtocolGenerator creates a new ProtocolGenerator object.
The method ~ProtocolGenerator terminates the ProtocolGenerator object.
A.1.2. Client
A.1.2.1. StreamingClient
Fig A.6: The StreamingClient class
79
This class is part of the StreamingClient Package. StreamingClient represents the
DataTransport module on the client and handles requests that are directed to the module.
Classes from other modules only need to know the StreamingClient class.
The startReception method starts a new process for the reception of data from the server.
The stopReception method ends the reception process for a specific data source.
The StreamingClient method creates the StreamingClient object.
The StreamingClient method terminates the object and closes all data receptions processes.
A.1.2.2. DataStream
Fig A.7: The DataStream class
The DataStream class is part of the TransportManagement Package. The codec and the compression ratio to be used only depend on decisions made at the server. So DataStream only has to react to changes of the Server. To prevent unnecessary overhead a
DataStream object at the client can change the used Codec.
The setCodec method changes the codec that the DataStream object uses.
The DataStream method creates a DataStream object.
The DataStream method terminates the DataStream object.
A.1.2.3. ServerManager
Fig A.8: The ServerManager class
80
The ServerManager class is part of the TransportManagement Package. ServerManager handles the communication with the server. The RTSP protocol is used to send requests to the streaming server, For each new data reception process ServerManager class creates a corresponding DataStream object.
The method startReception starts a new process for the reception of data from the server.
The method endReception ends the reception process of a specific data source.
The method handleRTCP method informs the ServerManager of a received
RTCPPacket.
The method ServerManager creates the ServerManager object.
The method ServerManager ends the ServerManager object and all ongoing data reception processes.
A.1.2.4. RTCPClient
Fig A.9: The RTCPClient class
RTCPClient is part of the TransportManagement Package. This class handles the exchange of RTCP packets with the server.
The method sendRTCP sends a RTCPPacket to the server.
The method addProtocol adds a RTPProtocol to the RTCPClient object. The
RTCPClient transmits information of all the RTP protocols it knows about.
81
The method removeProtocol removes a RTPProtocol.
The method RTCPClient creates a RTCPClient object.
The method RTCPClient terminates the RTCPClient object.
A.1.2.5. Codec
Fig A.10: The Codec class
The Codec class is part of the Codec Package. The Codec class is an interface that represents all codecs on the client.
The de c ompress method decompresses the data stored in memory.
The Codec method creates a Codec object.
The Codec method ends the compression process and terminates the Codec object.
A.1.2.6. Protocol
Fig A.11: The Protocol class
The Protocol class is part of the Protocol Package. The Protocol class is an interface that represents all protocols on the client.
The startReception method starts the data reception of the object. After a call to this method the Protocol object listens continuously for incoming data.
The endReception method ends the data reception.
The Protocol method creates a Protocol object.
The Protocol method ends the data transmission process and terminates the Protocol object.
A.1.2.7. CodecGenerator
82
Fig A.12: The CodecGenerator class
The CodecGenerator class is part of the Codec package. CodecGenerator is responsible for the creation of new codec objects.
The method enumCodecs returns a list of the supported codecs.
The method createCodec creates a Codec object. The new codec is returned to the caller.
The method CodecGenerator creates a CodecGenerator object.
The method CodecGenerator terminates the CodecGenerator object.
A.1.2.8. ProtocolGenerator
Fig A.13: The ProtocolGenerator class
The ProtocolGenerator class is part of the Protocol package. ProtocolGenerator is responsible for the creation of new codec objects.
The method enumProtocols returns a list of the supported protocols.
The method createProtocol creates a new Protocol object. The new Protocol is returned to the caller.
The method ProtocolGenerator creates a ProtocolGenerator object.
The method ProtocolGenerator terminates the ProtocolGenerator object.
A.2. Sequence Diagrams
A.2.1. HandleRTCP
Fig A.14: The HandleRTCP sequence diagram
83
Global
The client periodically sends an RTCP packet to the server that contains the statistics of the data reception. The RTCPServer receives this packet and delivers it to the
ClientManager. The ClientManager then reevaluates the transports with the client and may change some of the DataStreams that are used. This process shall now be described in detail.
Detail
When the RTCPServer receives an RTCP packet a corresponding RTCPPacket object is created. The ClientManager is then informed of the packet with the call to the method handleRTCP . In response the ClientManager retrieves the statistical information of the
RTCPPacket object.
In this case the ClientManager decides that a different compression ratio is required for one of the streams and requests this to the corresponding DataStream object. The
84
DataStream object changes the ratio and informs all the ClientManagers of the change by calling the notifyOfChange method for each ClientManager.
For another data transmission the ClientManager decides to join a different DataStream.
First it selects an appropriate DataStream. The ClientManager requests a list of
DataStreams for a particular data source to the DataStreamList object with a call to the method enumStreams .
The settings of these DataStream objects are investigated. After choosing a more suitable DataStream the ClientManager disconnects from the original
DataStream ( leaveStream ) and joins the selected DataStream ( joinStream ).
A.2.2. Schedule
Fig A.15: The Schedule sequence diagram
The TaskScheduler continuously decides which DataStream object or ClientManager may run next. In this case the DataStream object is activated first with the call to the method process . The activated DataStream objects instructs the Codec object to compress the data compress and then successively orders all the protocols to transmit transmit the data.
One of the ClientManager objects is activated next with the call to the method process .
An activated ClientManager may reevaluate the status of the client and connection and change compression ratios and DataStreams. The ClientManager uses this opportunity to send statistics to the Client with RTCP by calling sendRTCP to the RTCPServer.
85
Appendix B – Implementation classes
B.1. Class correlations
Fig B.1: The relations of the RTPStream class
RtpStream is responsible for the data transmission on the server and data reception on the client. This is done through the RTP protocol. Four different tasks are needed in the data processing. DataHandler for memory access, Codec for compression/decompression,
Rtptransmitter/Rtpreceiver for packing/unpacking the data and Transportprotocol for reception/transmission. These tasks are added to the stream in the open method of
Rtp_Stream.
Fig B.2: The relations of the RTCP_Stream class
Rtcp_Stream handles the exchange of feedback between server and client with the RTCP protocol. Each Rtcp_Stream handles one Rtp_Stream for one client.
RtcpTransmitter is used by Rtcp_Stream to create RTCP packets. RtcpReceiver is used to handle rtcppackets and TransportProtocol to transmit and receive these packets.
Fig B.3: The relations of the Codec class
86
Codec is an abstract class that represents all codecs. When data is send codec compresses data, at reception the data is decompressed. The two classes that inherit from Codec,
IjgReceiver and IjgTransmitter are responsible for respectively decompression and compression in the JPEG format using the IJG library.
Codec has a DataProperties object that contains the properties of the pictures such as width, height and bit depth.
Fig B.4: The relations of the TransportProtocol class
TransportProtocol is an abstract class that represents all transportprotocols. Classes that inherit from this class can be used for reception and transmission of data according to some protocol such as TCP and UDP.
A TransportProtocol object waits passively for arrival of data. The object is informed by ACE_Reactor when data arrives and then retrieves this data. ACE_Reactor is an ACE class that reacts to events on the system by informing the appropriate handlers. This requires a continuous call to the reactor method handle_events . An event handler has to be registered with the reactor before it can be used.
TransportProtocol therefore inherits from ACE_Event_Handler. The registry of a
TransportProtocol object has to be done by the stream using that object.
Why was this passive approach chosen? It would have been possible to simply attempt to retrieve data directly and wait until something arrives. This was the first used approach.
When however multiple transportprotocols are used to receive data the number of lost packets increases considerably. The reason for this is because only one object can listen for data in a single thread approach. In case of two transport protocol objects data will be intended for the object that is not listening in ½ of the time.
87
This might be prevented by creating a process or thread for each receiver, but when this was attempted an even worse performance was achieved even though all packets did arrive. Which is probably due to the extra required overhead.
The chosen solution does not lose packets (except for transmission loss on network/Internet) and is relative simple.
UDP inherits from TransportProtocol and handles data transmission and reception with the UDP protocol.
Fig B.5: The relations of the RtpTransmitter class
RtpTransmitter only handles the transmission of data and implements send of
Protocol_Task to do so. The data contained in the argument message is packed in one or more Rtp Packets that are passed on to the next task for transmission. Depending on the maximum allowed data size on the network the data is fragmented over multiple packets.
Packets belonging to the same data have the same rtp timestamp. Furthermore the last packet of the data gets a marker that identifies the packet as the last in the sequence.
The class RtpPacket is used as RTP header of the data. Because data compressed with jpeg requires an additional extension of the headers a class jpegExtension was created to form the basis of that extension. RtpPacket is a class of Vovida, an open source community.
RtpTransmitter is an adapted version of the Vovida class RtpTransmitter.
Fig B.6: The relations of the RtpReceiver class
88
RtpReceiver handles the reception of RtpPackets. Because data might be fragmented the packets have to be stored until the data is complete. The Frame class was created for that purpose. Packets that have the same Rtptime stamp are added to the same frame. The class jpegExtension contains the relative position of the data which helps in storing the data in the correct order in memory.
Once the first and last packet of a frame have arrived the frame might be complete. All packets have a successive sequence number so the number of received packets can be compared with the number of expected packets. Once the frame is complete it is placed in a queue. At a constant interval the oldest frame of this queue is passed on to the next
Protocol_Task in the stream (Codec). This also applies if the queue becomes to large.
RtpReceiver is an adapted version of the Vovida class RtpReceiver.
Fig B.7: The relations of the DataHandler class
DataHandler is responsible for handling the memory storage of data. In case of reception of data the data is stored in memory. At transmission the data is retrieved from memory.
DataHandler is used in Rtp_Stream and is the endpoint in case of data reception or the start point in case of transmission.
Fig B.8: The relations of the RtcpTransmitter class
RtcpTransmitter is responsible for creating RTCP packets that contain feedback regarding the transmission progress. On the server the RtcpTransmittter object knows one
RtpTransmitter, which is used to collect the transmission statistics such as the number of send packets. One RtcpReceiver object is also known by RtcpTransmitter for similar purpose.
RtcpTransmitter is an adapted version of the Vovida class RtcpTransmitter.
Fig B.9: The relations of the RtcpReceiver class
89
RtcpReceiver handles incoming rtcp packets. The class contains one RtpReceiver that is used to access reception statistics.
RtcpReceiver is an adapted version of the Vovida class RtcpReceiver.
B.2. Classes
B.2.1. ClientStream
Fig B.10: The ClientStream class
ClientStream represents a data stream on the client.
ClientStream : Creates a ClientStream object. The argument data is the memory destination of the pictures. The type of codec to be used for decompression is passed with cType . sendRtcp : Sends feedback to the server with rtcp. Returns 0 on success, -1 on failure.
Open : Opens ClientStream. Once opened data can be received from the server and Rtcp packets can be send to the server. A transportprotocol of type pType will be used to receive data at port rtpport . The clientstream will listen at port rtcpport for rtcp info from the server. The ipaddress serverip and port srv_rtcpport are used for sending back feedback to the server with rtcp.
B.2.2. Codec
Fig B.11: The Codec class
90
Codec : Creates a codec object. The properties of the data are provided with the argument dataInfo_ . setRatio/getRatio : Are used to set and get the compression ratio respectively.
B.2.3. CodecGenerator
Fig B.12: The CodecGenerator class
CodecGenerator generates codec objects. Only a single instance of this class will be needed so a class method instance has been created which can be used in other classes.
CodecGenerator : Creates a CodecGenerator object. createCodec : Creates a codec object of type ctype . If compression is true this codec is used to compress, otherwise the object is used for decompression. The argument prop are the properties of the data to be processed by the codec.
Instance : Returns a pointer to a process wide CodecGenerator.
B.2.4. Data
Fig B.13: The Data class
This class represents the memory allocated for visual data.
91
Data : Creates a Data object. The argument data_ is a pointer to the memory reserved for the data, prop is the properties of the data. getData : Returns a pointer to the memory that is reserved for this data. getProperties : Returns the properties of the data.
B.2.5. DataHandler
Fig B.14: The DataHandler class
DataHandler : Creates a DataHandler object. The argument _data is the buffer that has to be used for storage and retrieval of data.
Send : This method was inherited from Protocol_Task and stores the data contained in message in the buffer.
Recv : This method was inherited from Protocol_Task. Retrieves the data stored in the buffer and passes it on to the next task on the stream. The argument message is not used in this method but is required as argument.
B.2.6. DataProperties
Fig B.15: The DataProperties class
This class contains the properties of visual data.
DataProperties : Creates a DataProperties object. The argument dataType is the type of data represented by the object.
92
getDataType : returns the type of the data (FCCD, SCCD, RT) getWidth : Returns the width of the picture in number of pixels. getHeight : Returns the height of the picture in number of pixels. getBitDepth : Returns the number of bits per pixel. getSize : Returns the required number of bytes for a single picture with these properties. nrOfBytes : Returns the number of bytes required per pixel.
B.2.7. Frame
Fig B.16: The Frame class
Frame represents a single frame. RtpReceiver uses this class to store incoming data until all packets belonging to a single frame have arrived.
Frame : Creates a Frame object. The argument RtpTime is the RtpTime of the packets belonging to this frame.
Add : Adds a packet to this frame. The argument packet contains the header, data the body, size is the size of the body in bytes. The argument ext is the rtpheader extension for jpeg. Returns 0 on success, -1 on failure. getFirstSeq : Returns the sequence of the first known packet of this frame. getLastSeq : Returns the sequence of the last known packet of this frame. getRtpTime : Returns the RtpTime of this frame. isComplete : Returns true if this frame is complete, all packets have been added, false otherwise.
93
getData : Returns the framedata. getNrOfPackets : Returns the number of packets that have been added to this frame. getSize : Returns the size of the frame data in number of bytes.
B.2.8. IjgReceiver
Fig B.17: The IjgReceiver class
This class decompresses JPEG pictures with the IJG library and inherits from Codec.
IjgReceiver : Creates a IjgReceiver object. The argument dataInfo is the properties of the data that is to be processed by this object.
Recv : This method is inherited from Codec. The picture in message will be decompressed. The decompressed picture is then passed on to the next task object.
Returns 0 on success, -1 on failure.
Decompress : This method performs the actual decompression. The JPEG picture in compData is decompressed and stored in decompData . The argument size is the number of bytes in compData . Returns the number of bytes in the decompressed picture.
B.2.9. IjgTransmitter
Fig B.18: The IjgTransmitter class
This class compresses pictures to the JPEG format using the IJG library and inherits from
Codec.
IjgTransmitter : Creates a IjgTransmitter object. The argument dataInfo is the properties of the data that is to be processed by this object.
Send : This method is inherited from Codec. The picture in message will be compressed to
JPEG and passed on to the next task object on the stream.
94
Compress : This method performs the actual compression. The picture in rawData is compressed and stored in compData . Returns the number of bytes of the compressed picture.
B.2.10. jpegExtension
Fig B.19: The jpegExtension class
This class represents an extension of the rtp header for jpeg data, see RFC 2035 for further details regarding this extension. jpegExtension : Creates a jpegExtension object. Width is the width of the picture in pixels, height the height of the picture in pixels and offset is the offset of the data in this packet in the entire picture.
B.2.11. ProtocolGenerator
Fig B.20: The ProtocolGenerator class
ProtocolGenerator generates TransportProtocol objects. Only a single instance of this class will be needed so a class method instance has been created which can be used in other classes.
ProtocolGenerator : Creates a ProtocolGenerator object. createProtocol : Creates a TransportProtocol object of type ptype , which listens at port port for incoming data.
95
Instance : Returns a pointer to a process wide ProtocolGenerator.
B.2.12. Protocol_Stream
Fig B.21: The Protocol_Stream class
Protocol_Stream represents a stream in which the data flows through several protocol layers (or filters). Specialized stream classes inherit from this class.
Protocol_Stream : Creates a Protocol_Stream object.
Close : Close the stream. All of the tasks & modules will also be closed.
Send(message) : putting data onto the stream will pass it through all protocol levels and send it to the peer.
Send : Calls send(message) with an empty message.
B.2.13. Protocol_Task
Fig B.22: The Protocol_Task class
This class represents a task in a stream. Classes that inherit from Protocol_Task specialize in a certain protocol and either implement the send and/or the recv method.
Protocol_Task : Creates a Protocol_Task object.
Open : is invoked when the task is inserted into the stream.
Close : is invoked when the stream is closed.
Put : As data travels through the stream, the put() method of each task is invoked to keep the data moving along. This method is called by put_next.
96
Process : Called by put() as necessary to process a block of data.
Send : Tasks on the writer (upstream) side of the stream are called upon to send() data.
This method will be implemented in inherited classes that process data that flows through the stream upstream. Returns 0 on success, -1 on failure.
Recv : Tasks on the reader (downstream) side will be receiving data. This method will be implemented in inherited classes that process data that flows through the stream downstream. Returns 0 on success, -1 on failure.
B.2.14. RtcpReceiver
Fig B.23: The RtcpReceiver class
Only the main methods are listed.
RtcpReceiver : Creates a RtcpReceiver object.
Recv : Retrieves the statistics contained in the RtcpPacket in message . Calls readSR to handle a RtcpPacket. isValid : Returns true if p is a valid RTCP packet. This is a Vovida method readSR : Reads the contents of head . Is called by recv .
B.2.15. RtcpStream
Fig B.24: The RtcpStream class
97
Rtcp_Stream : Creates a Rtcp_Stream object
Open : Opens the Rtcp_Stream object. Once opened the object can be used for transmission and reception of feedback. The stream shall listen at recvport for incoming
RTCP packets. In case of transmission of feedback RTCP packets are send to IP address ip and port sendport . A protocol of type pType (for instance UDP) is used for transmission and reception of these packets. This method returns 0 on success and -1 on failure.
Close : Closes the Rtcp_Stream. Once called the Rtcp_Stream will no longer listen for incoming RTCP packets and can not be used for transmission of such packets. This method may only be called if the object is open.
Returns 0 on success, -1 on failure. setRtpTran, getRtcpTran and getRtcpRecv are used to connect the Rtcp_Stream with the
Rtp_Stream of which it provides feedback.
B.2.16. RtcpTransmitter
Fig B.25: The RtcpTransmitter class
Only the main methods are listed.
RtcpTransmitter : Creates a RtcpTransmitter object.
Send : This method was inherited from Protocol_Task and creates a RtcpPacket that is passed on to the next task for transmission. The argument message is not used.
Returns 0 on success, -1 on failure.
98
setRTPtran,setRTCPrecv are used to set the objects that RtcpTransmitter retrieves statistics from. addSR : Is a Vovida method that is used in send to create a RtcpPacket.
B.2.17. RtpReceiver
Fig B.26: The RtpReceiver class
To limit the complexity of RtpReceiver only the new methods and attributes are listed.
The Vovida methods and attributes are not shown.
RtpReceiver : Creates a RtpReceiver object.
Recv : Unpacks and processes the RTP packet contained in message . The method handlePacket is called by Recv to handle a valid RTP packet. This method returns 0 on success, -1 on failure. handlePacket handles a decomposed RtpPacket. The argument p contains the standard headers of the packet, ext contains the extension headers for JPEG data and data contains the body of the packet.
HandlePacket tries to find the corresponding frame for the packet and if no such frame exists a new frame object is created. Incomplete frames are stored in the vector frames.
Once a frame is complete addCompleteFrame is called to place the Frame in the vector receiveQueue where it will await further processing on the stream. addCompleteFrame puts frame in receiveQueue which is kept in order of age of the frames.
If the queue becomes too large (> MAXRECQUEUESZ) the oldest frame is removed from the queue and passed on to other Tasks in the stream by calling handleFrame . Frames are also passed on to other tasks at a constant interval.
HandleFrame passes frame on to the next task and removes all incomplete frames that are older than frame to maintain the correct order in time of frames.
99
setRTCPRecv sets the RtcpReceiver that corresponds to object. This method is called to connect the Rtp_Stream to the corresponding Rtcp_Stream.
B.2.18. RtpStream
Fig B.27: The RtpStream class
Rtp_Stream : Constructs a Rtp_Stream object. The object will use data as the memory buffer for visual data. In case of reception data is the endpoint of data, in case of transmission it is the source of data. The argument cType is the type of codec to be used for compressing or decompressing data, for instance IJG.
Open : Opens the Rtp_Stream for transmission or reception. The direction of the dataflow, transmission or reception, is indicated with the argument transmode . The type of protocol to be used for the data transfer on a network or the Internet is provided with pType , which can be for instance UDP. If Rtp_Stream is used for reception the object will listen at port for incoming data. This method returns 0 on success, -1 on failure.
Close : Closes the Rtp_Stream. Once called the Rtp_Stream will no longer listen for incoming data and can not be used for data transmission. This method may only be called if the object is open.
Returns 0 on success, -1 on failure.
AddReceiver/RemoveReceiver : Add and remove a receiver. Rtp_Stream will send data to all receivers that have been added. setRatio/getRatio : Set/get the used compression ratio. getRtpRecv/getRtpTran : These methods return the used RtpReceiver, RtpTransmitter object respectively. These methods allow ClientStream/ServerStream to connect
Rtcp_Stream objects with the Rtp_Stream object.
B.2.19. RtpTransmitter
100
Fig B.28: The RtpTransmitter class
MTU : The maximum allowed transport size in bytes (Maximum Transport Unit).
RtpTransmitter : Creates a RtpTransmitter object. DataProperties are the properties of data that is handled by this object.
Send : Is inherited from Protocol_Task. Packs the data in message in one or more
RtpPackets and passes these on for transmission. Returns 0 on success, -1 on failure.
B.2.20. ServerStream
Fig B.29: The ServerStream class
ServerStream represents a data stream on the server.
ServerStream : Creates a ServerStream object. The argument data is the memory source of the pictures. The type of codec to be used for compression is passed with cType .
Open : Opens the ServerStream object. Once opened the stream can be used for transmission of data and exchange of feedback with clients. The protocoltype to be used for transmission and reception is pType .
Returns 0 on success, -1 on failure.
101
Close : Closes the stream. Once closed ServerStream no longer can be used to send data or feedback to the clients. Neither shall incoming Rtcp packets be handled by
ServerStream. Returns 0 on success, -1 on failure. sendData : Sends the picture that is in the buffer to all clients. Returns 0 on success, -1 on failure. sendRtcp : Sends an RtcpPacket containing the latest statistics of the transmission (such as number of send packets) to all clients. Returns 0 on success, -1 on failure. addClient : Adds a client to the stream with address ip , that listens at port rtpPort for data and at port clnt_rtcpport for rtcp info. The serverstream will listen at port rtcpport for incoming rtcp packets from this client. Returns 0 on success, -1 on failure. removeClient : Removes the client with address ip . setRatio,getRatio are used to set and get the compression ratio of the used codec.
B.2.21. TransportProtocol
Fig B.30: The TransportProtocol class
TransportProtocol : Creates a TransportProtocol object which listens at port for incoming data. addReceiver / removeReceiver : Add/Remove a receiver. getHandle,handle_input,handle_close : These methods are required for ACE_reactor and are inherited from ACE_Event_Handler. Handle_input is called by ACE_Reactor when data arrives.
Send : Is inherited from Protocol_Task. Sends data contained in message to all receivers.
102
