INF5070 – Media Storage and Distribution Systems: Introduction and Architectures 29/8 - 2005 Overview Intro about the course multimedia applications and challenges Architectures Media (Video) on Demand Machine internals Video server structures Examples INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen INF5070: The Course Lecturers Carsten Griwodz email: griff @ ifi office: 3343 Pål Halvorsen email: paalh @ ifi office: 3345 INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Content architectures Networ k file systems Networ k media data distribution Networ k resource scheduling Networ k protocols topologies INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Content - lectures System architectures (server and system designs) Media and user behavior (wire and file formats, codecs, processing multimedia data, user interactivity) Resource management (CPU and memory management) Protocols with and without Quality of Service (QoS) (specific and generic QoS approaches) INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Content - lectures Storage systems (management of multimedia files, retrieval) Distribution (use of caches and proxy servers) Peer-to-Peer (various clients, different amount of resources) Guest lecture 7/11: The FAST searching system (architecture: resource utilization and performance, storage and distribution of data, parallelism, etc.) INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Content - student assignment Mandatory student assignment (will be presented more in-depth later): write a project plan describing your assignment (3/10) write a report describing the test and its results, and give a presentation to the class at the end of the course (28/11) for example (examples from earlier): TCP friendliness variations New TCP implementations Network emulators Comparison of Linux schedulers File system benchmarking (different OSes and file systems) Comparison of open source video servers (Helix vs. Darwin) Comparison of methods for network performance monitoring … INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Goals Media servers and distribution system media characteristics architectures system support protocols distribution mechanisms … Be able to evaluate any combination of these mechanisms, e.g., data placement on disk and memory caching algorithm video popularity and proxy caching algorithm … INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Exam Prerequisite: approved presentation of student assignment Oral exam (??/12-2005): all transparencies from lectures content of your own student assignment INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Applications and Challenges Applications Multimedia enriches the user interface giving new applications Broadcast – server is only VCR substitute (e.g., MTV Europe) Media-on-Demand (MoD) Network-based, user controlled distribution of media content, e.g.: Video-on-Demand (VoD) (e.g., Bell Atlantic (‘93)~1000 users, 700 choices) News-on-Demand (NoD) (e.g., CNN, BBC web today, BAD quality) Learning-on-Demand (LoD) (e.g., the OMODIS project) Virtual worlds (typically live, not on-Demand) video conferencing (e.g., USIT’s electronic classroom) games (e.g., Anarchy online, Quake, …) INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Requirements Application QoS – time sensitivity resource capabilities – support interactive streaming of multimedia content Business scalability reliability Architectural topology cost vs. performance INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Technical Challenges User end system real-time processing of streams (1000 MIPS for an MPEG-II decoder) request/response delay (< 150 ms for videophones) high data rates, e.g., MPEG-II DVD quality: average video rate of 3.5 Mbps average transport stream of 4 – 8 Mbps (video, audio, headers, error protection) max. total video data rate of 10.08 Mbps max. user rate of 11.08 Mbps (all included like control signals) Network real-time transport of continuous media data TCP fairness mobility … INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Technical Challenges Servers (and proxy caches) storage real-time retrieval of contiguous media streams, e.g.: o o I/O * 15 Mbps (HDTV) = 40.5 TB * 1.4 Mbps = 1.4 TB many concurrent clients expecting continuous playout of DVD-quality video (~ 4Mbps) o o o 4000 movies * 90 minutes 2000 CDs * 74 minutes disk: Seagate X15 - ~400 Mbps network: Gb Ethernet (1/10 Gbps) bus(ses): - PCI 64-bit, 133Mhz (8 Gbps) - PCI-Express (2.5 Gbps each direction/lane) computing in real-time encryption adaptation transcoding … INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Media-on-Demand (MoD) Systems Classification parameters interaction structure of movies presentation form Common directions analog digital media distribution interaction media broadcasting media multicasting personalized media linear movies branched movies variable movies Evolution broadcast - traditional, no user control near MoD - same media distributed in intervals (pay-per-view) true MoD - full user control, VCR capabilities, bidirectional INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Television (Broadcast) channels time sender • analog or digital • traditionally, one program per channel receiver(s) analog use frequency division multiplexing only digital may additionally use time division multiplexing inside one frequency (several programs per channel) • linear movies INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Near Video-on-Demand (NVoD) channels time sender • analog or digital broadcasting • one program over multiple channels • time-slotted emission of the program • linear movies INF5070 – media storage and distribution systems receiver(s) 2005 Carsten Griwodz & Pål Halvorsen (True) Video-on-Demand (VoD) movies time sender receiver(s) • digital uni- or multicasting • control channels • linear movies INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Comparison: NVoD vs. TVoD Response Services Costs Suited NVoD TVoD delayed immediate provide some flexibility compared to traditional TV improves video rentals: any video, any time cheap: more clients at lower price compared to TVoD digital broadcast (ViaSat, TV1000) INF5070 – media storage and distribution systems expensive: existing infrastructure often not cost-competitive for households (need to be equal to rentals) specific (smaller) environments (hotels, airplanes, …, cities,…) 2005 Carsten Griwodz & Pål Halvorsen “Interactive Vision” movie time sender receiver(s) • digital uni- or multicasting • control channels • fixed non-linear movies INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen “Cyber Vision” time sender • digital uni- or multicasting • control channels • variable non-linear “movies”, e.g., - games, virtual reality, … INF5070 – media storage and distribution systems receiver(s) 2005 Carsten Griwodz & Pål Halvorsen Application Classification Overview Cyber Vision movie structure variable Interactive Vision branched HDTV VoD linear digital TV NVoD unidirectional VIDEO interaction INF5070 – media storage and distribution systems bidirectional analog 2005 Carsten Griwodz & Pål Halvorsen Media (Video) on Demand Challenges VoD goals: network-based distribution of media content to consumers bring control to users VoD in LANs is solved: OVERPROVISIONING works established in studio business established in small area (hotel/hospital/plane/…) businesses Network INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Challenges VoD in WANs assumptions: programs: need for interoperability – not from a single source need for co-operative distribution systems amount of data: overprovisioning of resources will NOT work no central control of delivery system estimated 65000 movies made in 1995 260 TB MPEG-2 data additionally, data from TV-series, sport clips, news, … historically: much attention as “interactive TV (ITV)” some years ago many (not successful) field trials now: interest turned to Internet-based systems INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen ITV Network Architecture Approaches WAN backbones SONET ATM Local distribution network ATM / SONET backbone network ADSL (asymmetric digital subscriber line) FTTC (fiber to the curb) FTTH (fiber to the home) HFC (hybrid fiber coax) (=cable modem) E-PON (Ethernet passive optical network) … wireless Internet based systems ADSL telephone Internet WAN Diffserv over MPLS (multi-protocol layer switching) point-to-point Gbps ethernet cable Internet local IP over the old distribution networks INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Concerns: Internet-Based VoD Systems Can technical problems be mastered? broadband communication to every home user-friendly end systems server technology Market success? what prices will consumers accept? will it be competitive? high equipment (HW & SW) costs data costs to existing TV programs to video rentals what is the consequence of ITV field trials in USA and Europe? no big success only a few private consumers willing to participate trials were cancelled or switched from TV-based to PC-based platforms INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Driving Forces Hardware/software (IT) companies computer (e.g., IBM, HP, Sun, Microsoft, …) consumer electronics (e.g., Sony, Philips, …) Network companies telephone (e.g., Telenor, Telia, BT, AT&T, …) cable TV (e.g., UPC, Time-Warner Cable, …) Content companies media movies (e.g., Time-Warner, Disney, …) TV programs (e.g., NRK, TV2, TV3, …) hyper media information bases (e.g., Springer, Bertelsmann, …) home-shopping (e.g., Amazon, Ebay, …) video games (e.g., Nintendo, Sega, Microsoft, …) INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen VoD Deployment Status – I Digital Video Broadcast (DVB) no VoD cable, antenna, or satellite broadcast some NVoD scheduling approaches (e.g., TV 1000) Digital Audio-Visual Council (DAVIC) defines interfaces only no standardization of algorithms for interoperation closed project Broadcast world suited for Near VoD Broadcast world suited for True VoD Internet world currently no large-scale, wide-area video distribution suited for True VoD Internet Engineering Task Force (IETF) sporadic use of cooperative web caching starting AV caching considerations defines protocols and inspires interoperability testing INF5070 – media storage and distribution systems MPEG 4 suited for Interactive Vision 2005 Carsten Griwodz & Pål Halvorsen VoD Deployment Status – II DATA CONTROL RTP: Real-Time Protocol RTSP: Real-Time Streaming Protocol package & timing information for controls: setup, teardown, start, stop IETF transfer profiles for each encoding format RTCP: RTP Control Protocol report exchange, allows tuning at SDP: Session Description Protocol carried by RTSP encoding information, timing, meta info the sender MPEG-2 Transport: DSM-CC: Distributed Storage Media- defines encoding format, packaging, MPEG-2 substandard: addressing, setup, Moving Pictures Expert Group DAVIC timing, scaling, error correction requires other means of addressing INF5070 – media storage and distribution systems Command & Control teardown, start, stop, … independent standard: complete management standard (usually ignored by industry) 2005 Carsten Griwodz & Pål Halvorsen VoD System Architecture backbone network local distribution network INF5070 – media storage and distribution systems local distribution network local distribution network 2005 Carsten Griwodz & Pål Halvorsen VoD Storage Hierarchy Use hierarchies completeness of available content master servers Popularity of movies: not all are equally popular – most request directed to only a few (Zipf distribution) regional servers Straight forward hierarchy: popular videos replicated and kept close to clients locality vs. communication vs. server costs INF5070 – media storage and distribution systems local servers end-systems 2005 Carsten Griwodz & Pål Halvorsen VoD Components Servers Networks backbone local networks Intermediate nodes routers proxy cache servers End-systems PCs TV sets with set-top boxes INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Traditional Server Machine Internals General Operating System Structure and Retrieval Data Path application user space kernel space file system INF5070 – media storage and distribution systems communication system 2005 Carsten Griwodz & Pål Halvorsen Example: Intel Hub Architecture (850 Chipset) – I Intel D850MD Motherboard: RDRAM connectors CPU socket system bus RDRAM interface hub interface PCI bus Memory Controller Hub I/O Controller Hub PCI connectors INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Example: Intel Hub Architecture (850 Chipset) – II Note: these transfers only show dataapplication movement between sub-systems and not the commands themselves. communication Additionally, data touching file systemoperations within a subsystem system will require that data is moved from memory and to the CPU, e.g.: disk - checksum calculation - encryption network card - data encoding - forward error correction Pentium 4 Processor registers cache(s) system bus (64-bit, 400/533 MHz ~24-32 Gbps) memory controller hub RAM interface (two 64-bit, 200 MHz ~24 Gbps) RDRAM file system RDRAM communication system RDRAM application RDRAM hub interface (four 8-bit, 66 MHz 2 Gbps) I/O controller hub PCI slots PCI bus (32-bit, 33 MHz 1 Gbps) INF5070 – media storage and distribution systems network card PCI slots PCI slots disk 2005 Carsten Griwodz & Pål Halvorsen Example: IBM POWER 4 application POWER 4 chip CPU L1 Note: Again, data touching file systemoperations add movement operations CPU L1 disk core interface switch communication system network card L2 fabric controller GX controller RAM file system communication system RAM application RAM PCI busses GX bus (two 32-bit, 600 MHz ~35 Gbps) remote I/O (RIO) bridge L3 L3 controller memory controller (32/64-bit, 33/66 MHz 1-4 Gbps) PCI host bridge PCI-PCI bridge PCI host bridge PCI-PCI bridge PCI slots network card PCI slots disk RIO bus (two 8-bit, 500 MHz ~7 Gbps) INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Example: AMD Opteron & Intel Xeon MP 4P servers application file system communication system disk network card Know your hardware – different configuration may have different bottlenecks INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Server Internals Data retrieval from disk and push to network buffer requirements bus transfers CPU usage concurrent streams (can be merged??) storage (disk) system: scheduling – ensure that data is available in time block placement – contiguous, interleaving, striping … Stable operations: redundant HW multiple nodes Much more, e.g., caching/prefetching, admission control, … INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Video Server Structure Video Server: Server Components & Switches Internal content directory: External content directory: HP, DEC, Novell, … IP, … network attachment RPC in application, … switch content directory/ memory management NFS, … switch HP, DEC, Novell, …. switched network switch file system AFS, CODA, … IBMswitch TigerShark switched network switch switch switch switch storage management distributed OS, … switch switch switch IBM TigerShark controller Disk arrays (RAID), … switch switch storage device INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Video Server: Simple General Server Architecture Storage subsystem: • stores data • different devices Processor subsystem: • executing part • management and operations Network subsystem: • transmit MM data storage subsystem data server control server network subsystem clients processor subsystem application server data control Data server: Application server: Control server: • data delivery • user interface • administrator • “specialized file system” • billing • admission control • buffer manager • content database • optimization • data importer/exporter • user database • service gateways A similar architecture is used in SGI/Kassena Media Base & IBM VideoCharger INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Video Server: Directory Access & Data Retrieval Two-step retrieval: data give “challenge”: permission resource management content Request redirection: “challenge”: client gets data from another machine INF5070 – media storage and distribution systems e.g., Real-time Transport Protocol (RTP), maybe RTSP Network e.g., Hypertext Transfer Protocol (HTTP), Session Initiation Protocol (SIP), maybe Real-Time Streaming Protocol (RTSP) data e.g., TCP handover or handoff Network content 2005 Carsten Griwodz & Pål Halvorsen Video Server: Directory Access & Data Retrieval Pull model: client sends several requests deliver only small part of data fine-grained client control favors high interactivity suited for editing, searching, etc. Push model client sends one request streaming delivery favors capacity planning suited for retrieval, download, playback, etc. INF5070 – media storage and distribution systems server client e.g., Priority Progress Streaming (PPS) server client e.g., HTTP, File Transfer Protocol (FTP), RTP + RTSP 2005 Carsten Griwodz & Pål Halvorsen Video Server: Server Topology – I Single server easy to implement scales poorly Network Partitioned server users divided into groups content : assumes equal groups location : store all data on all servers load imbalance INF5070 – media storage and distribution systems Network Network 2005 Carsten Griwodz & Pål Halvorsen Video Server: Server Topology – II Externally switched servers use network to make server pool manages load imbalance (control server directs requests) still data replication problems (control server doesn’t need to be a physical box - distributed process) data Network data data control data Fully switched server server pool storage device pool additional hardware costs e.g., Oracle, Intel, IBM I/O switch Network data control INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Video Distribution Server: Typical In the Internet Today Push systems (pull in video editing/database systems) Traditional (specialized) file systems – not databases – for data storage No in-band control (control and data information in separate streams) External directory services for data location (control server + data pump) Request redirection for access control Single stand-alone servers (fully) switched servers INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Server Examples Video Server Components Research incoming resolve request incoming data request delivered resolution delivered data network attachment content directory network attachment IBM Watson Tetzlaff, Kienzle, … memory management control/application server file system storage management controller storage device data server INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Video Server Components Research Härtig, Sitaram, Dan, Nahrstedt, Steinmetz, Klas, Shulzrinne, Coulson, Seltzer, Rangan, Zhang, Hutchinson, … incoming data request delivered data Goyal, Vin, Peterson, Tetzlaff, Kienzle, … network attachment Fall, Druschel, Pai, Silberschatz, Özden, Buddhikot, Miller, … memory management Shenoy, Nirajan, Silberschatz, Özden, Martin, Haskin, … file system Ghandeharidzadeh, Zimmermann, Haskin, … storage management Jones, Nieh, Chen, Berson, Reddy, Ghandeharidzadeh, Zimmermann, Haskin, … controller storage device data server INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Video Server “Product Status” 1) Real server, VXtreme, Starlight, VDO, Netscape Media Server, MS Media Server, Apple Darwin RTSP user level server RTP standard OS 2) IBM Mediastreamer, Oracle Video Cartridge, N-Cube all standard HW 3) SGI/Kassena Media Base, SUN Media Center, IBM Video Charger user level server RTSP RTP user level layer DSM CC, private ATM, analog scalable, RT-aware OS, RT OS, or OS derivation custom/special HW MM standard RT FS OS extensions selected standard HW INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Real Server User space implementation one control server several protocols several versions of data in same file adapts to resources request 3 Quality-of-Service load leveling INF5070 – media storage and distribution systems backpressure track 2 Does not support track 1 index user Real’s own kernel MPEG-2 version with “stream thinning” (dropped with REAL ) 1 Real’s protocol Several formats, e.g., server 2 TCP UDP RTP/ RTCP feedback IP 2005 Carsten Griwodz & Pål Halvorsen IBM Video Charger May consist of one control AIX SP2 crossbar switch machine only, or … … several IBM’s Advanced Interactive eXecutive (AIX) machines Servers control data Lightly modified existing components OS AIX4/5L virtual shared disks (VSD) VSD with (guaranteed disk I/Os) EDF specific control server RTSP video stream API distributed computing environment RPC mlib API filter encrypt RTP TigerShark MMFS UDP VSD IP Special components TigerShark MMFS (buffers, data rate, prefetching, codec, ...) stream filters, control server, APIs, ... DESCRIBE SETUP PLAY TEARDOWN INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen IBM Mediastreamer Version of Video Charger failed project to guarantee MPEG-2 over ATM and Cable similar machine setup as Video Charger (machine cluster) special HW: SCSI controller MPEG-2 ATM or analog cable out moved to Video Charger AIX SP2 crossbar switch control specific control server RTSP video stream API distributed computing environment RPC mlib API Unlike Video Charger, Mediastreamer runs on old IBM machines due to special HW Special components TigerShark control server APIs special board INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen n4x media hubs: • Intel 860 Chip Set • Intel 1.5 GHz Xeon CPU • Up to 2 GB Rambus Memory • Five 64 bit 66Mhz PCI slots • “Special” PCI slot (HIB board) • nHIO hypercube I/O nCUBE Original research from Cal Tech/Intel (‘83) Bought by C-COR in Jan. 05 (~90M$) One server scales from 1 to 256 machines, 2n, n [0, 8], using a hypercube architecture Why a hypercube? Highlights video streaming is a switching problem hypercube is a high performance scalable switch no content replication and true linear scalability integrated adaptive routing provides resiliency one copy of a data element scales from 5,000 to 500,000 clients exceeds 60,000 simultaneous streams 6,600 simultaneous streams at 2 - 4 Mbps each (26 streams per machine if n = 8) 8 hypercube connectors Special components boards with integrated components TRANSIT operating system n4 HAVOC (1999) Hypercube And Vector Operations Controller ASIC-based hypercube technology n4x nHIO (2002) configurable interface SCSI ports nCUBE Hypercube I/O controller (8X price/performance) INF5070 – media storage and distribution systems memory PCI bus vector processor 2005 Carsten Griwodz & Pål Halvorsen nCUBE: Naturally Load-balanced Disks connected to All MediaHubs Each title striped across all MediaHUBs Streaming Hub reads content from all disks in the video server Content striped across all disks in the n4x server Automatic load balancing Immune to content usage pattern Same load if same or different title Each stream’s load spread over all nodes RAID Sets distributed across MediaHubs Immune to a MediaHUB failure Increasing reliability Only 1 copy of each title ever needed Lots of room for expanded content, network-based PVR, or HDTV content INF5070 – media storage and distribution systems Video Stream 2005 Carsten Griwodz & Pål Halvorsen nCUBE: Sample deployments & Cost TWC Los Angeles – South Bay HE Monterey Park CA 2040 streams from 18 n4 MediaHUBs West Covina CA 840 streams from 8 n4 MediaHUBs Pasadena CA 840 streams from 8 n4 MediaHUBs Glendale CA 1320 streams from 12 n4 MediaHUBs Long Beach CA INF5070 – media storage and distribution systems 1320 streams from 12 n4 MediaHUBs 2005 Carsten Griwodz & Pål Halvorsen Small Comparison Real Video Charger nCUBE standard HW selected HW special HW each machine its own storage, or NFS shared disk access, no replication shared disk access, no replication single OS image cluster machines using switch cluster machines using wired cube user space server user space server and loadable kernel modules server in both kernel and user space (except for load leveling and fault tolerance) INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen Summary Multimedia applications and challenges Media (Video) on Demand Machine internals Video server structures Video server examples INF5070 – media storage and distribution systems 2005 Carsten Griwodz & Pål Halvorsen