http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Multimedia-Systems:
Multimedia Hardware Computer Architectures
Prof. Dr.-Ing. Ralf Steinmetz
Dr. L.Wolf, Dr. S.Fischer
TU Darmstadt - Darmstadt University of Technology,
Dept. of Electrical Engineering and Information Technology, Dept. of Computer Science
KOM - Industrial Process and System Communications, Tel.+49 6151 166151,
Merckstr. 25, D-64283 Darmstadt, Germany, Ralf.Steinmetz@KOM.tu-darmstadt.de Fax. +49 6151 166152
GMD - German National Research Center for Information Technology
IPSI - Integrated Publication and Information Systems Institute, Tel.+49 6151 869869
Dolivostr. 15, D-64293 Darmstadt, Germany, Ralf.Steinmetz@darmstadt.gmd.de Fax. +49 6151 869870
07-hardware.fm 1 22.October.99
Usage
Services
Systems
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Scope
Applications
Learning & Teaching
Content
Processing
Documents
Design
Security
Opt. Memories
Group
SynchroCommuninization
cations
...
Databases
Media-Server
User Interfaces
Programming
Operating Systems
Communications
Quality of Service
Networks
Basics
Compression
07-hardware.fm 2 22.October.99
Computer
Architectures
Image &
Graphics
Animation
Video
Audio
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Contents
1. Motivation
2. Computer Components
3. Computer Architecture:
4. Towards Integrated Digital Systems
07-hardware.fm 3 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
1. Motivation
Multimedia systems comprise:
• Hardware and software components
• Local and distributed systems
Hardware is base of all multimedia efforts:
• e.g. tremendous driving impact of CD-ROM technology
Problems of traditional approaches:
• Existing technology (RISC processors, signal processors, etc.)
not yet exploited by operating systems or other software components
• Network technology not integrated in multimedia systems
Technology is a necessary,
but not a sufficient condition
for multimedia.
07-hardware.fm 4 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
2. Computer Components
Multimedia computer:
• Designed for simultaneous manipulation
of discrete and continuous media information
Components:
• Processors:
• CPU(s) for processing discrete (and continuous) media
• universal signal processors
• processors dedicated to graphics, audio or video data processing
• Storage:
• primary and secondary
• Adapters:
• communication
• graphics, audio, video devices
• input/output
• Connections:
• busses, switches, interfaces
07-hardware.fm 5 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Computer Components: Processors
CPUs and DSPs:
• Monoprocessor system (CISC, RISC technology)
• Dedicated special processors like DSPs (Digital Signal Processors)
• Advanced: multiprocessor combination of DSPs and CPUs
• if possible, on single chip
CPU features for multimedia support:
• High clock speed
• Capabilities for high data throughput
• see above
• Multiple subsystems executing in parallel
• e.g. math coprocessor, image coprocessor, MPEG decoder, communication
block
• Extended instruction set
• specific instructions needed for multimedia data processing
• e.g. for image compression / decompression, graphic manipulation
07-hardware.fm 6 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Computer Components: Storage
Primary storage:
• SRAM (static RAM)
• DRAM (dynamic RAM)
• ROM
• PROM
• EPROM
• Standard computer technology, but large primary storage capacities
necessary,
typically 32 MB RAM minimum for desktop multimedia applications
Secondary storage:
• Hard- or floppy disks as carrier possible, but limited capacity
• CD-ROM
• Guaranteed continuous data transfer
• Important role of controller (see busses)
07-hardware.fm 7 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Computer Components: Multimedia In-/Output Devices
Audio:
• Microphone, headphone
• Loudspeakers (active, passive)
Video:
• Camera
• Monitor: high resolution, color, tube or LCD monitor, flat, big
Other media:
• Text, graphics, stills, animation etc.
• Usual computer equipment with high quality standard
• keyboard, mouse, scanner, printer, graphic tools, bitmap editors etc.
07-hardware.fm 8 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Computer Components: Busses
System busses:
• Well-known technology on system bus: IBM Microchannel, EISA, Q-Bus,
VME-Bus, PC-Bus, Apple NuBus, NeXT NeXTBus
• Asynchronous bus for normal data traffic, time guarantees on a second
bus
Peripheral busses:
• ESDI
• SCSI:
• Asynchronous: about 1.5 MByte/s
• Synchronous: about 5 MByte/s with 8 bit in parallel
• SCSI-2: about 40 MByte/s with 32 bit in parallel
Alternative, network instead of bus:
• ATM switch as internal interconnect of computer components
• Offers various parallel data paths
07-hardware.fm 9 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Computer Components: Interfaces
Audio:
• Existing audio technology
• Examples: Consumer standards (Cinch, DIN), professional interfaces
(XLR)
Video:
• Existing video technology
• Examples: NTSC, PAL, SECAM, FBAS, YUV, YIG
Other media:
• Existing computer technology
• Serial and parallel interfaces
• Examples: DB-25, DB-9, Centronics
07-hardware.fm 10 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Further Multimedia Hardware Components
• Network adapters
• Coprocessors
• Interrupt controllers
• DMA controllers
• Timers, clock generators
07-hardware.fm 11 22.October.99
timesharing
CPU(s)
asynchronous bus
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
3. Computer Architecture:
Conventional Computer:
communication port
CPU(s)
graphics/video
CPU(s)
disk & memory
CPU(s)
07-hardware.fm 12 22.October.99
interconnect (bus)
timesharing
CPU(s)
communication port
CPU(s)
graphics/video
CPU(s)
audio/video/image
CPU(s)
real-time
CPU(s)
disk & memory
CPU(s)
07-hardware.fm 13 22.October.99
real-time local interconnect
(crossbar or bus)
Advanced architecture: an example
asynchronous local
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Computer Architecture: Advanced
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Computer Architecture: Internal Interconnects
Crossbar switches:
• Multiple connections between pairs of components
for parallel communication
ATM switch:
• Used for internal interconnection of computer
system components
• Connect components directly to high-speed ATM network
• Integrates external and internal network
• „Everything uses ATM cells“
• Same protocol techniques
Network as interconnect for computer components:
• Internal network instead of bus
• No translation between multiplexing techniques used on bus and on
external network
07-hardware.fm 14 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Computer Architecture: Desk Area Network
DAN (Desk Area Network) as multimedia workstation:
• Switch used to deliver AV streams directly to relevant devices
Network
Synchronisation
server
Frame Grabber
or ATM Camera
07-hardware.fm 15 22.October.99
Frame Store
CPU
DAN
Audio Server
Decompression
Compression
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
For Furher Studies Example Processors: Sun UltraSPARC
UltraSPARC architecture:
• 64-bit processing
• Special support for
• high data throughput
• multimedia-specific operations
• parallel instruction execution
• „Allows to handle 2 MPEG-2 video streams in parallel to other tasks“
(according to Sun Microsystems)
Used in Sun’s Ultra workstations
07-hardware.fm 16 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
UltraSPARC: Data Throughput
Ultra Port Architecture (UPA) interconnect:
• To connect subsystems on main board
• CPU, memory, graphics subsystem, I/O interfaces, ...
• Crossbar switch
• potentially combined with bus
• Data packets of various subsystems can be interspersed
• no subsystem dominates the communication link
-> different from bus-based CPUs
• Data rates of up to 1.3 GBytes/s
(according to Sun Microsystems)
• Error correction code (ECC) for all data transfers
Wide memory interface:
• 256 bits for data
• plus 32 bits for ECC
07-hardware.fm 17 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
UltraSPARC: Data Throughput (cont.)
SBus extension bus:
• 64-bit bus
• To connect main board to other cards
• e.g. disk controllers, network controllers
• Up to 123 MBytes/s
Fast SCSI interface:
• Up to 20 MBytes/s
Network interfaces:
• Fast Ethernet controller on motherboard
• Additional extension cards available for
• ATM
• ISDN
• FDDI
07-hardware.fm 18 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
UltraSPARC: Multimedia Data Processing
Visual Instruction Set (VIS):
• Multimedia-specific instructions, e.g. for
• image manipulation
• audio and video compression / decompression
• 2-D / 3-D graphics
• SIMD operations:
• „Single Instruction Multiple Data“
• e.g. 32-bit word including multiple bytes for color (R,G,B) and intensity of a
pixel
• single VIS instruction can manipulate the whole word,
i.e. operate on values in parallel
Graphic and multimedia data processing shared between:
• CPU
• Graphic subsystem
• on extra card
• with „3D-RAM memory“
• memory with special circuitry to accelerate graphic processing
07-hardware.fm 19 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Intel MMX
MMX = Multimedia Extension
• Enhancement of Intel Pentium processor
• integrated into Pentium chip
• To accelerate multimedia and communications applications
Main features:
• Enhanced instruction set
• 57 special instructions for multimedia data processing
• design based on analysis of multimedia software applications
• SIMD architecture
• execution of the same operation on multiple data in parallel
• e.g. on eight bytes packed into a 64-bit quadword
• Eight 64-bit registers
07-hardware.fm 20 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
4. Towards Integrated Digital Systems
Stages of evolution:
1. Separate analog and digital components
2. Computer controls analog audio/video components
3. Digital, but not integrated communication
4. Integrated communication,
but separate local AV components
5. Digital systems with analog interfaces
6. Digital systems with all digital interfaces
07-hardware.fm 21 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Stage I: Traditional Approach
Separation of analog and digital data:
• Continuous vs. discrete media
• Analog vs. digital
• Telecommunications vs. computers
digital
• mention: no interaction
07-hardware.fm 22 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Stage II: Computer Controls External Devices
Make use of:
• AV components
• Computers control analog devices:
• e.g. via SCSI , RS-232C, other interfaces
Examples:
• Integrated Media Architecture Laboratory (IMAL), Bell Research
• Video Services in Muse and Pygmalion of the MIT Athena projects
• Starting point for DiME project, IBM ENC, Heidelberg
control
digital
07-hardware.fm 23 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Stage III: Digital Communications
Digital, but not integrated communication:
• Digitalization of continuous media
• Separate net access
• Example:
• US West in Boulder: DS3-connection used for communication of two
multimedia laboratories
control
digital
07-hardware.fm 24 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Stage IV: Integrated Digital Communications
Integrated communication, separate Local AV Components:
• Only some continuous media data stream routed through the computer
• Use of available AV technology
• Example:
• Etherphone developed by XEROX PARC
with integrated communication
but, e. g. separate storage
control
digital
07-hardware.fm 25 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Stage V: All Digital But Analog Interfaces
Digital systems with analogue interfaces:
• All data routed through the computer
• Examples:
• Etherphone at Xerox PARC: Audio communication over Ethernet
• AT&T in Naperville: Fast packet-switching net attached to workstation:
• Expansion of UNIX towards “connectors”, “active devices”
• DASH in Berkeley
• HeiTS with e. g. DVI at IBM ENC
digital
07-hardware.fm 26 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Stage VI: All Digital System
Digital systems with all digital interfaces:
• Assumes corresponding terminal equipment
• Advantage to previous architecture:
• Error free reproductions
digital
07-hardware.fm 27 22.October.99