CS3162 Introduction to Computer Graphics
Helena Wong, 2000
11. Image Processing
Image processing concerns about modifying or transforming images. Applications may include
enhancing an image or adding special effects to an image. Here we will learn some of the image
processing functions. The description will be based on grey-scale images. However, extending the
techniques to color images are usually straight-forward.
Histogram Modifications
-
A histogram is a plot of the frequency of
occurrence, p(g), against the gray level
value, g.
-
The distribution of gray scale values
affects the quality of the image. The
following diagram shows some examples:
(a) If most of the gray level values in an image are small, the image will be dark.
(b) If most of the gray level values in an image are large, the image will be bright.
(c) If most of the gray level values in an image are within a small region, the image will have a
low contrast. Therefore, both (a) and (b) are also considered as low contrast images.)
(d) A high contrast image has a wide range of gray level values.
-
Modifying the histogram by changing the frequency of occurrence of each gray scale value may
improve the image quality and enhance the contrast.
-
Brightness Correction is used to increase the brightness of a dark image as follows:
Pixel(x,y) = Pixel(x,y) + brightness
Brightness correction is therefore equivalent to shifting the histogram horizontally.
-
Contrast Correction is used to modify the contrast of an image as follows:
Pixel(x,y) = contrast * (Pixel(x,y) – gmean) + gmean
Where gmean is the mean gray scale value of the original image.
Hence contrast correction is equivalent to scaling the histogram. We can reduce the contrast of an
image by setting the factor contrast in the above equation to a value smaller than 1 and we can
increase the contrast by setting it to a value higher than 1.
1
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
Editing of Intensity / Color of an Image
-
Very often, it may be desirable to be able to edit the
gray scale values in a linear as well as non-linear way.
-
To do this, a curve is usually provided which indicates
the relationship of the original and new values.
-
This curve can be edited interactively by moving the
control points.
-
A color image is composed of red, green and blue channels.
-
Three curves are usually provided so that each of the channels can be modified independently.
Warping
-
Warping is the use of image mapping functions to produce
geometric distortions of images.
-
The basic technique is that the user specifies the contour of the
object in the image to be transformed and the final contour that
the object is to fit into.
-
A texture mapping operation is performed to map the source
object into the destination shape.
Morphing
-
Morphing is used to provide a smooth transition from one image to another and thus create an
illusion of image transformation.
-
A simple morphing may use a fade or dissolve process that gradually decreases the contribution
of the source image to the output image while increases the contribution of the destination image
as follows:
-
Depending on applications, the decrement (or increment) in contribution of the source (or
destination) image into the final image may not have to be linear as in the diagram.
-
More advanced morphing techniques may use a warping process in addition to the fade / dissolve
to slowly bring into alignment the shapes of the start and end pictures.
2
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
12. Window Systems
A window system manages a computer screen and divides the computer screen into overlapping
regions. Each region displays output from a particular application. Since the X window system is
widely used in most of the Unix machines, we will use the X11 window system as a basis for this
discussion.
The Goals of X
-
To provide low-level graphics workstation services
-
-
windows
2-D color bitmap graphics
multiprocess access to workstation screen
low-level input processing
To have local-area network transparency
-
applications on same or different network node of the workstation
portability of application software
multiple displays
open architecture
-
To provide standard programming interfaces, eg. “Xlib” library interface
-
To provide user interfaces
- “xterm” terminal emulator
- other tools such as editors and calculators
- programming toolkits
-
To encourage further development
- Inexpensive, widely available source code
Features of the X11
-
It is a hardware and operating system independent window system developed jointly by MIT and
DEC.
-
It is based on a client-server model.
-
Multiple clients can be connected to a display server simultaneously.
-
The X11 provides programmer interfaces through Xlib and X Toolkits.
-
Xlib provides basic functions for drawing and for manipulating the windows. Eg.,
XcreateWindow(display, parentwin, x,y,width,height,..);
XSetLineAttributes(display, gc, linewidth, linestyle, ..);
XdrawLine(display, win, gc, x1,y1,x2,y2);
-
X Toolkits provide a higher level of interface to Xlib through widget. A widget is a routine for
creating and using user-interface components. A widget set may include menus dialog boxes
scrollbars and command buttons.
3
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
Server and Client
The server is a program running on the workstation. It acts as an intermediary between user programs
(or clients) and the resources of the workstation such as keyboard, mouse, and screen. It contains all
device-specific code so that client applications do not need to know the hardware of the server
resources. The server performs the following tasks:
-
Allows access by multiple clients.
-
Allows client applications to share resources such as screen space and graphics processor.
-
Interprets network messages (or requests) and acts on them. Requests include “move window”
and “draw polygon”.
-
Sends user inputs to clients by sending network messages, eg. key presses, button presses, and
pointer motion.
-
Maintains complex data structures including windows and fonts so that the server can perform its
tasks efficiently.
In X window system, the term display is often used to refer to server. The term screen refers to the
physical hardware screen and a display may have more than one screen.
A Client is an application which requests services from the server. A client may or ma not be running
on the same machine as the server (network transparency). All communication between a client and a
server uses the X Protocol.
Window Manager
In X11, window manager is just another client. However, there can only be one window manager
managing a particular screen at any one time. The tasks of the window manager include:
-
Controls the behavior of the screen such as the functionality of each mouse button.
-
Controls the layout of the windows such as title bars and borders.
-
Controls the size and displacement of windows.
-
Manage icons.
The following diagram shows the structure of the X11 window system:
4
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
Windows and Window Hierarchy
The following diagram shows the dimensions of a window;
Windows in X are organised in a hierarchical manner.
-
The top window, which covers the whole screen surface, is called root window and is opened
automatically.
-
A window will obscure any overlapping windows that are behind it.
-
Any sub-windows are bounded by its parent window.
-
Any part of a sub-windows goes outside of its parent window will be clipped.
-
Any thing to be drawn but are outside of the specified window will be clipped too.
-
Since rectangle is easy to do clipping, most window systems use rectangle as the basic window
shape.
-
Although a window may originally be rectangular, it may become irregular when it is obscured by
other windows (eg. window A in the previous diagram).
-
In order to simplify the clipping operation, the X approach is to break an obscured window into
rectangular windows as shown in the following diagram:
-
In the diagram, window B is divided into 3 rectangular regions a, b, and c. Clipping an object
against Window B can now be done by clipping the object against each of the 3 rectangular
regions and display any parts of the object which are inside any of the regions.
5
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
13.Virtual Reality (VR)
There are many definitions for the term Virtual Reality. The following definition is from the book
Silicon Mirage by Steve Aukstakalnis and David Blatner:
Virtual Reality is a way for humans to visualize, manipulate and interact with computers and
extremely complex data. … Methods for human-computer interaction are called computer
interfaces, and virtual reality is just the newest in a long line of interfaces.
General speaking, virtual reality has the following features:
-
It emphases in the interaction with the computer through gestures and human senses.
-
Instead of using screens and keyboards, people can put displays over their eyes, gloves on their
hands, and headphones on their ears.
-
The computer controls what the user senses and the user, in turn, can control the computer.
-
With such an interaction, the user may have a feeling of entering into a different world (virtual
world).
-
Instead of entering into a different world, sometimes, we may want to be able to obtain interactive
information related to the existing world. We call this Augmented VR.
Levels of Virtual Reality
There are 3 levels of VR. Each of them offers slightly different features and each progressive step is
much more difficult to create.
-
Passive: In passive VR, we do not have much control over the environment that we are
experiencing. It includes watching a movie or TV show, reading a book, and listening to the
radio. Taking reading as an example. While we may be able to choose a particular book to read,
we cannot change the story of it.
-
Exploratory: The 2nd level of VR is that of being able to explore a virtual world. Instead of just
seeing a3D space, we can move around in it, whether by flying or walking.
-
Interactive: The 3rd level of VR is that of being able to explore and interact with a virtual world.
We may do this by reaching out and grabbing a virtual book, or moving furniture around in a
virtual room.
The term “VR” normally refers to either 2nd level or 3rd level of VR.
6
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
Human and Computer Communications
Human communicate through variety of senses. Similar kind of communication may be necessary in
order for a person to communicate with a computer in the same way as to another person.
-
Vision: Visual effects play a very important role in human communication.
-
A lot of research has been done on improving the realism of the computer generated images
and the speed of generating them so that the computer can present its own world to the user.
-
In the same way, a lot of research has been done on improving the computer vision
technology so that the computer can understand the user’s world through vision. (This has
proven to be far more difficult than the previous one.)
-
Sound: Audio can provide a direct way for human to exchange ideas. It also helps a person to
sense its surrounding, eg. the presence of an approaching car and its approximate distance.
Similar to vision, it is easier to generate sound than to recognise it.
-
Other Senses: Other senses include smell and touch.
-
To produce different kind of smell as well as to identify them are both difficult to do. These
factors are rarely considered.
-
There are existing equipment which can detect forces from or exert forces onto a user.
However, this kind of equipment is usually intrusive in that the user can feel their presence
even through they are not in operation at a particular moment. Eg. one can feel the presence
of the glove when he/she is wearing it.
Types of VR Systems
A major distinction of VR systems from traditional computer systems is on how the user
communicates with the computer. There are many types of VR systems developed for different kinds
of applications. Four of them are Immersive VR, Non-immersive VR, Augmented VR, and
Telepresence.
Immersive VR
The ultimate VR system is to completely immerse the user to a computer simulated environment.
-
Because visual effects play such an important role in human communication and 3D graphics
hardware is getting cheaper and cheaper, existing immersive VR systems emphasis on visual
immersion.
-
A head Mounted Display (HMD) with position tracker is usually used to provide a virtual 360°
vision.
-
Some systems may use multiple large projection screens to visually surround the user.
-
Other systems may provide even the same physical environments. These are usually limited to
applications such as flight simulators and car simulators in which the users are confined to a small
room.
7
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
Non-immersive VR
Unlike the immersive VR, non-immersive VR systems do not provide a 360° of vision and the user is
not required to wear an HMD or be inside a simulation room. Hence, non-immersive VR is also
called “limited reality”.
-
Most low cost systems use traditional monitors for visual output.
-
Some systems may use an additional equipment (such as the
CrystalEyes 3D stereo glasses) to provide stereo vision. In the
CrystalEyes glasses, the 2 lenses allow light to pass through
alternatively in synchronization with the displayed images to provide
stereo vision. These systems are slightly more expensive than those
using traditional monitors along and they require the user to wear a
pair of glasses.
With these systems, the user is inside the virtual world only when he is
looking at the computer screen. As soon as he looks away from it, he is
immediately back to the real world.
Augmented VR
Augmented VR is the use of transparent glasses onto which data, diagrams, animation or video can be
projected so that the user is able to see the real world with additional information provided by the
computer.
-
There are 2 common types of displays available. One is a pair of glasses. A small projector is
attached to it so that video images can be projected onto the pair of glasses.
The other one is the flight helmet also used by the pilots of the fighter jets. A small projector is
attached to the helmet so that additional information from the computer about the environment
can be projected onto the transparent screen.
Telepresence
Telepresence links the sensors and manipulators of a remote robot to a human operator.
communication is in both ways.
The
-
Cameras, microphones and other sensors are usually attached to the remote robot. Signals from
these sensors are sent to the human operator in a control room. The operator may be wearing an
HMD and other devices which render the remote signals.
-
The human operator is also wearing some sensors which sense the movement of the operator.
The signals are sent to the remote robot to control its movement.
This technology is very useful in performing some dangerous operations such as putting out the fire in
a nuclear plant, or in performing some otherwise too expensive operations such as Mars exploration.
8
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
Position Sensors
In order to explore or to interact with a virtual world, it is important for the computer to know the
user’s physical coordinate. The following are some of the methods:
-
2D mouse: While 2D mouse is always available, it is difficult to use in 3D world. In addition, it
only returns relative (not absolute) movement of the mouse.
-
Mechanical tracker: A mechanical device is attached to the object. The movement of the object
results in the movement of the mechanical device, which generate the positional signal. This kind
of devices are fast and accurate but has restrictions on motion.
-
Ultrasonic tracker: An ultrasonic transmitter is attached to the object to be tracked and a few
sensors are placed at different positions. By detecting the time it takes for the sensors to receive
the ultrasonic signal from the transmitter, it is possible to locate the position of the object. (Some
products may use a few transmitters and a single sensor to detect the location of an object.)
However, ultrasonic systems are of low resolution and are susceptible to noise.
-
Light pattern: By putting a pattern of light transmitters to the object and cameras to the wall (or
vice versa), it is possible to locate the position of the object by detecting the patterns of light
captured by the cameras. Limitations are that it requires the use of the image processing
technology, which Is not reliable enough, and a dedicated room.
-
Magnetic tracker: Magnetic tracing is one of the widely used methods.
-
Basically, a magnetic transmitter is placed at a fixed location and a magnetic sensor is
attached to the object.
-
Both the transmitter and sensor contain 3 coils at right angles to each other.
-
The position and orientation of the sensor affect the amount of current induced to the sensor
by the transmitter.
-
As such, the sensor can be used to detect the 3D position (x,y,z) and orientation of an object.
-
Very often, time multiplexing is used so that more than one sensor can be connected to a
single transmitter system.
-
Advantage: Magnetic tracking devices are cheap and accurate.
-
The major limitation of magnetic tracking device is that the accuracy of these devices is
affected by nearby metallic materials.
-
Another limitation is that it is generally believed that strong magnetic fields are harmful to
human body.
A Typical Immersive System
The configuration of a typical immersive VR system includes a Head Mounted Display (HMD), an
electronic glove, a set of position tracking devices and a graphics workstation.
-
The HMD contains a pair of LCD screens and a pair of optical lenses. Together they provide the
user with stereo-vision. The HMD also contains a pair of speaker to provide stereo sound.
-
An electronic glove is a glove with a lot of sensors around it to detect the bending of fingers.
-
Magnetic systems are commonly used for position tracking.
9
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
-
Normally, a magnetic sensor will be attached to the HMD and another sensor will be attached to
the electronic glove so that the positions and orientations of the user’s head and hand can be
detected.
-
The graphics workstation is expected to generate stereo images in real-time, which are to be sent
to the 2 display screens inside the HMD.
-
As we move our head, the 2 display screens will show images corresponding to what we are
supposed to see at each particular head position.
-
As we move our hand, the computer will also know where our hand is in 3D space so that we can
grab objects inside the virtual world.
Time Lag
All VR systems suffer from the time lag problem. Consider the following VR system:
-
When we move our head, the magnetic sensor senses the change and generates a signal. There is
a delay from we move our head to the generation of the signal.
-
When the head tracking system receives the signal from the sensor, it processes the signal to
obtain the new position and orientation of the sensor. There is also a delay in this process.
-
When the tracking system sends out the information, the computer receives the information,
process it and then updates the database. There is a delay in this process.
-
After the database is updated, the image generator renders an updated image from the updated
database. There is also a delay in this process.
-
Hence from the time the head moves to the time the image on the screen is updated constitutes to
the total system lag.
-
This time lag causes an uncomfortable feeling to the user because we are used to an instant visual
response in real life.
-
Some work is being done to reduce such problem by predicting what may the head be few steps
ahead. This method of course has its own problem as the prediction may not always be correct.
10
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
Virtual Reality vs Computer Graphics
-
In computer graphics, we are concerned about the quality of the output image and we may also be
concerned about the speed in generating the image.
-
In VR, we are also concerned about the quality of the images. However, with the limitation in
current hardware technology, we are more concerned about the speed in generating the images.
-
In VR, in order to give the user an interactive feeling, we must be able to generate at least 10
images per second and the time lag problem must be within an acceptable level.
-
To be able to generate at least 10 images per second, time critical rendering becomes an
important issue of study: given the time we have, how can we generate the best picture possible.
-
Eg., if time is not enough, we may render objects with flat shading instead of Gouraud shading.
-
We may also use simpler models for rendering less important objects.
-
Another issue of study is image coherence.
-
Since the time between 2 consecutive frames is very small, the change in the scene is small too.
-
Hence, given the information we have about the scene from the previous frame, we can make use
of it and minimize the amount of calculation in generating the next image.
-
Eg., if an object is far away from the viewer in the last frame, it is likely that it is still far away in
the current frame.
Current Areas of Research
-
Multi-resolution modelling: This uses a simpler model for rendering when the object is far away
from the viewer. This is to reduce the number of polygons needed to be rendered.
-
Fast rendering of surfaces: Usually curve surfaces are broken down into polygons for rendering.
This area tries to speed up the polygonization process.
-
Fast rendering of deformable surfaces: Again, this area tries to speed up the polygonization
process of deforming surfaces.
-
Collision detection: This detects if 2 objects collide with each other.
-
Visibility culling: This determines the visibility of a surface or an object.
Applications of the VR Technology
-
Surgical training
Architectural walk-through
Telepresence
To assist the disables to communicate.
Visualization
VR games
11
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
13. Multimedia
Multimedia is the integration of media technologies including sound, video, document, network and
graphics with the computer technology. There are 2 major areas of interest in multimedia:
-
Multimedia technology: the technology itself.
Multimedia applications: the application of the technology
Because multimedia covers a wide range of disciplines, we will only briefly look at this technology.
Multimedia Technology
The integration of media technologies may be considered as putting all the devices that handle
different media together with the computer being the central controlling device as shown in the
following diagram:
However, when the multimedia workstation is connected to a high-speed network, it is possible to
have most of the video and audio services provided through the network as in the following diagram:
12
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
-
These systems allow us to watch TV, to have video-conferencing with someone in remote
location, to read an electronic document, etc...
-
In a networked multimedia system, all these multimedia services can be provided by some central
servers so that any computers in the network can have access to these services. In such a
situation, a multimedia computer only needs to be equipped with a video interface to display /
capture real-time images and an audio interface for sound input and output.
-
Under such a situation, we can also have a large central database to provide variety of information
services.
One of the most demanding media technologies is the handling of video. Consider color images of
video quality – 512x512 in resolution:
-
The memory size needed to store a single image, Msingle, is:
Msingle = 512 x 512 x 3 Bytes = 0.75 M Bytes
-
Assuming that we have 30 frames per second. The memory size needed to store a second of the
image sequence, Msec is:
Msec = 0.75 M Bytes x 30 = 22.5 M Bytes
-
A movie typically lasts for 90 minutes. The memory size needed to store the whole movie,
Mmovie, is:
Mmovie = 22.5 M Bytes x 60 x 90 = about 118.7 G Bytes
For color images of movie quality, eg. resolution of 1600 x 900 (16:9) with 24 frames per second:
Msingle = 1600 x 900 x 3 Bytes = 4.12 M Bytes
Msec = 4.12 M Bytes x 24 = 98.88 M Bytes
Mmovie =98.88 M Bytes x 60 x 90 = about 521.4 G Bytes
With such a huge amount of memory, handling of videos digitally becomes a driving force to improve
other technologies such as database (to store the images), network (to transfer the images from the
server to the computer) and data compression (to compress / uncompress the images before / after
transmission or storage). Other related issues include the demand for high resolution displays.
Related Technologies
1. Display Technology:
With windowing systems being widely used in both PCs and workstations, and the consumer
community pushing for high definition television (HDTV), display technology is advancing rapidly.
Monitors of resolution 1600x1280 are commonly available. At the same time hardware circuitries are
also available which can read the frame buffer fast enough to update the screen near to 100 times a
second.
2. Network Technology:
A local computer network connects many computers together so that a small community can share
resources such as disk space, printing facilities, and processing power. With most of the local
networks in the whole world being connected together through the INTERNET, it is now possible to
share some of the resources in a global scale. A fast Ethernet network for video transmission is
required. Recent developments in this area include the 100 Mbits/s fast Ethernet network and the over
1 Gbits/s fiber optical network.
13
CS3162 Introduction to Computer Graphics
Helena Wong, 2000
3. Database Technology
In a multimedia environment, there may be many different types of data that conventional data
management systems are not prepared to handle. The data objects can span an enormous range of size
and format. Such a multimedia database contains a mixture of structured data (relations with fixed
fields) and of unstructured data (text and images). All these factors demand for new models of data
management systems which can provide the typical join and union operations for structured data
queries, and unstructured text, image, and sound queries. Because mot multimedia systems are built
using an object paradigm, the new models must also provide object oriented facilities for handling of
data objects. All these issues are in various stages of research.
4. Image Compression
The needs for fast network to transmit digital videos and for large database to store them demand for
better and faster image compression methods.
5. Sound Technology
There are two major issues: music synthesis and speech processing.
Music synthesizing technology is becoming matured. In addition, the protocol called MIDI is also
widely accepted by manufacturers as a standard for networking musical instruments.
Speech technology can be divided into 2 categories: Speech synthesis and Speech recognition
6. Memory Technology
A multimedia system may need to have access to a large volume of information. This demands for
more RAMs and secondary storage. With the continuous improvement in memory technology, both
RAMs and secondary storage devices are getting cheaper and cheaper.
Multimedia Applications
Applications of the multimedia technology are continuously to be proposed and developed. Some of
them are:
1. Video conferencing: allows 2 or more people at different physical locations to communication
with each other via voice and video through telephone or high speed networks.
2. Video services: are usually provided to users who have their machines connected to a high speed
network. In a cable network environment, subscribers receive different video services from a
central server(s) maintained by the cable company.
3. Video shopping: is a kind of services usually provided in a cable network environment in which
the user can purchase some products simply by pressing a button on the remote controller while
watching an advertisement on the cable TV.
4. Interactive environment for collaboration: is the use of multimedia technology to provide an
interactive environment for collaboration through work space sharing, and video and audio
communication.
5. CD-ROM publishing: is the distribution of CDROMs which store the interactive multimedia
catalogues or interactive books.
14