Implementing Video Systems Using the Altera Video Framework © 2011 Altera Corporation—Internal AGENDA What is the Video Framework? Overview VIP Core Details Boards/Reference Designs © 2011 Altera Corporation—Internal 2 Altera Video Framework Building block IP cores to speed up development (VIP suite) Open, low-overhead interface standard to mix-and-match custom or off-the shelf IP blocks Format conversion reference designs that showcase silicon capability, provide a starting point for design System Level Tools and design methodology Variety of development boards/kits for rapid design prototyping © 2011 Altera Corporation—Internal 3 The Video IP framework – The IP Building block IP to speed up development Interfaces (SDI, DVI, etc), High Performance Memory Controllers, Nios Video IP suite: library of interoperable Video IP blocks using common standards and protocols Open interface standard to mix-and-match custom or offthe shelf IP Avalon Memory-Mapped and Avalon-Streaming Video protocol Open Source HDL Template to facilitate integration of custom IP into framework Avalon-ST Video (over Avalon-ST) Input interface Clocked Video Input Altera VIP 1 Nios Video in © 2011 Altera Corporation—Internal 4 Avalon-MM Custom IP Altera VIP 2 Clocked Video Input Output interface Memory controller Video out The Video IP framework – Design Tools SOPC Builder Quartus II 1080p reference designs: out-ofthe-box ASSP-level capabilities and/or great starting points V-series, M-series, partner designs Pin mapping, Interface IP, Top-level HDL Abstract design capture Switch fabric generation IP Catalogue tool Nios II IDE Nios II IDE / Quartus II Software library to control the cores of the VIP Suite and for interrupt servicing Development boards/kits for rapid design prototyping Debug (Nios II & SignalTap) © 2011 Altera Corporation—Internal 5 Stratix II GX, Cyclone III Stratix IV GX PCIe, Arria II GX Software control and configuration AGENDA What is the Video Framework? Overview VIP Core Details Boards/Reference Designs © 2011 Altera Corporation—Internal 6 Video Image Processing (VIP) Suite BT 656 Avalon ST video Scaler v10.1 Chroma resampler Deinterlacer v11.0 Deinterlacer II Switch 2D FIR filter Control Sync Frame buffer 2D median filter Frame Reader Image clipper Color space converter Avalon ST video BT 656 Gamma correction Test pattern generator Color plane sequencer Scaler II Alpha blending mixer Half a dozen hardware verified video processing reference designs using these functions © 2011 Altera Corporation - Public 7 Available TODAY! New Cores in VIP Suite (2011) Scaler II Reduces area and improves performance Compared to first-generation scaler Supports 4:2:2 chroma data sampling rate Further reduces area used Deinterlacer II Supports 3:2 cadence detection Supports low-angle edge detection Delivers lower latency © 2011 Altera Corporation - Public 8 VIP Function Details Chroma Resampler This function allows you to change between 4:4:4, 4:2:2 and 4:2:0 sampling rates For 4:2:2 4:4:4, the filtered algorithm uses a 4-tap filter with fixed Lanczos-2 coefficients For 4:4:4 4:2:2, the filtered algorithm uses a 7-tap filter with fixed Lanczos-2 coefficients © 2011 Altera Corporation—Internal 9 VIP Function Details Color Space Converter This function provides a flexible and efficient means to convert image data from one color space to another You can select from a list of common color space conversion Or choose a custom color space © 2011 Altera Corporation—Internal 10 VIP Function Details Deinterlacer / Deinterlacer II (motion adaptive) Select the input image height and width Choose the deinterlacing algorithm – bob, weave, motion adaptive The weave and motion adaptive algorithms require external frame buffering © 2011 Altera Corporation—Internal 11 Understanding Algorithmic Constraints Deinterlacer (motion adaptive) Motion adaptive deinterlacer requires five (master) accesses to the DDR memory 4:4:4 Mode, Motion Bleed is ON 1 field write 2 field reads 1 motion vector write 1 motion vector read Deinterlacer (MA) Calculating DDR memory bandwidth Input format: 1080i, 60 fields/sec, 10-bit color 1920 × 1080 × 30bits × 60/2 = 1.866Gbit/s Output format: 1080p, 60 frames/sec, 10-bit color 1920 × 1080 × 30bits × 60 = 3.732Gbit/s Motion format: Only use 10bits for the motion values 1920 × 1080 × 10bits × 60/2 = 0.622Gbit/s 1 × write at input rate: 1.866Gbit/s 1 × write at motion rate: 0.622Gbit/s 1 × read at motion rate: 0.622Gbit/s 2 × read at output rate: 7.464Gbit/s Total: 10.574Gbit/s © 2011 Altera Corporation—Internal 12 DDR2 Memory access: 4:4:4 Mode Understanding Algorithmic Constraints Deinterlacer (motion adaptive) Motion adaptive deinterlacer requires five (master) accesses to the DDR memory 1 field write 2 field reads 1 motion vector write 1 motion vector read 4:2:2 Mode, Motion Bleed is ON Deinterlacer (MA) Calculating DDR memory bandwidth Input format: 1080i, 60 fields/sec, 10-bit color 1920 × 1080 × 20bits × 60/2 = 1.24Gbit/s Output format: 1080p, 60 frames/sec, 10-bit color 1920 × 1080 × 20bits × 60 = 2.48Gbit/s Motion format: Only use 10bits for the motion values 1920 × 1080 × 10bits × 60/2 = 0.622Gbit/s Memory access: 1 × write at input rate: 1.24Gbit/s 1 × write at motion rate: 0.622Gbit/s 1 × read at motion rate: 0.622Gbit/s 2 × read at output rate: 4.96Gbit/s Total: 7.44Gbit/s © 2011 Altera Corporation—Internal 13 DDR2 4:2:2 Mode, 30% reduction in DDR bandwidth VIP Function Details Requires progressive 4:4:4 input Scaler Select the input and output image sizes Choose the scaling algorithm – number of taps/phases Choose from a selection of pre-built filter functions or choose a custom function © 2011 Altera Corporation—Internal 14 VIP Function Details Scaler II – Utilizes Fewer Resources Reduces area and improves performance Compared to first-generation scaler Supports 4:2:2 chroma data sampling rate Further reduces area used Algorithm supports Linear and polyphase with run-time coefficient load New core in HDL More efficient in using resources © 2011 Altera Corporation - Public 15 VIP Function Details Alpha Blending Mixer Select the size of the image Select the number of layers to be mixed The level of blending is controlled by the number of alpha bits (up to 8) © 2011 Altera Corporation—Internal 16 VIP Function Details Generate a color bar test pattern © 2011 Altera Corporation—Internal 17 BT656 Avalon ST Video The Clocked Video Input MegaCore function converts from clocked video formats (such as BT656 and DVI) to Avalon-ST Video It strips the incoming clocked video of horizontal and vertical blanking, leaving only active picture data No conversion is done to the active picture data The color plane information remains the same as in the clocked video format. © 2011 Altera Corporation—Internal 18 BT656 Avalon ST Video The Clocked Video Output MegaCore function converts from Avalon-ST Video to clocked video formats (such as BT656 and DVI) It formats Avalon-ST Video into clocked video by inserting horizontal and vertical blanking and generating horizontal and vertical sync information No conversion is done to the active picture data The color plane information remains the same as in the Avalon-ST Video format. © 2011 Altera Corporation—Internal 19 Frame Buffer © 2011 Altera Corporation—Internal 20 Run-time Control of the Video Chain Build video signal chains that can be updated onthe-fly without changing the HDL or bit-stream Using the streaming interface protocol: Avalon ST Video Using an embedded processor © 2011 Altera Corporation—Internal 21 Run-time Control of the Video Chain Video function 1 Video function 1 Avalon ST READY DATA VALID SOP EOP Examples of Avalon-ST Video Control Packets © 2011 Altera Corporation—Internal 22 Avalon-ST Video protocol is a packet-oriented way to send video and control data First packet contains a frame of video data Second packet contains control information that applies to the subsequent video packet/frame Run-time Control of the Video Chain Video function 1 Avalon MM control plane Avalon MM Master port Processor Most video functions permit run-time control of some aspects of their behavior, using a common type of AvalonMM slave interface Each slave interface provides access to a set of control registers which must be set by an embedded processor/logic Generally, a function can be updated on a frame boundary © 2011 Altera Corporation—Internal 23 NIOS Control: Cyclone III Example Design Initialize daughter card Initialize daughter card Start the IP cores Control the mixer and Scaler parameters at runtime © 2011 Altera Corporation—Internal 24 AGENDA What is the Video Framework? Overview VIP Core Details Boards/Reference Designs © 2011 Altera Corporation—Internal 25 Video Development Kits Cyclone III FPGA, $1,895 Cyclone IV FPGA, $1,295 Stratix IV FPGA, $4,995 Arria II GX FPGA, $2,995 $495 $795 Cyclone III FPGA, $2,995 © 2011 Altera Corporation—Internal 26 New Demo in 2011 (VEEK) Demo # 1 – Simple Video Format Conversion Input Processing composite de-interlaced video /w txt overlay (NTSC to 800x600 with real-time resizing, zoom and panning) Output VEEK display VGA out Demo # 2 – Camera Input Scaling Input Processing Camera on VEEK Camera Input/w txt overlay (Camera input scaling with real-time resizing, zoom and panning) Output VEEK display VGA out Download Design Files on Altera Video Wiki www.alterawiki.com/wiki/Videoframework © 2011 Altera Corporation—Internal 27 New Demo in 2011 (VEEK) Demo # 3 – Picture in Picture (PiP) Input Processing Output composite camera on VEEK de-interlaced video /w txt overlay PIP - (camera source) VEEK display VGA out Demo # 4 - Multi-view Input Processing Output composite camera (GPIO) DVI (HSMC) CH 1 / 3: de-interlaced video CH 2 : camera video CH 4: DVI video (800x600 down scaling) DVI (HSMC) @ 800x600 Watch Demo of VIP Suite © 2011 Altera Corporation—Internal 28 New Low Cost UDX Reference Design C4GX150 board Develop in Quartus II software v11.0 using Qsys Available at end of July 1 channel UDX design SDI / DVI input switch SDI / DVI output Resolution supported 1920 x 1080P 50/59.94/60Hz 1280 x 720P 50/59.94/60Hz Text overlay and testing pattern © 2011 Altera Corporation—Internal 29 Video Reference Designs Design Device Features I/Os Video and image processing (VIP) demo Cyclone III, Cyclone IV National Television System Committee (NTSC)-to-480p scaling, 720p camera input scaling, mixing for PiP, multiview , 1-channel UDX design Analog, digital video interface (DVI) Cyclone® III FPGA design Cyclone III NTSC-to-720p scaling, MA deinterlacer, scaler, mixing for PiP Analog, DVI UDX2 Arria® II GX Same as UDX4 Serial digital interface (SDI), DVI UDX3/4 Stratix® IV GX V1, on-screen display (OSD), switch, active format description (AFD), mixing, multiport front-end (MPFE), 1080p, 720p, 1080i, ntsc/pal, 4:2:2 scaler, MA deinterlacer with low-angle edge SDI, DVI, high-definition multimedia interface (HDMI) 4K scaling Stratix IV GX 1080p 4K scaling, component design methodology SDI 4x SDI for output Broad Portfolio of Reference Designs to Help You Get Started © 2011 Altera Corporation—Internal 30 Summary Altera video framework enables rapid development Mix and match existing IP – Leverage Altera’s open interface standard Automatically integrate embedded processors and arbitration logic Leverage building block IP provided by Altera Use existing reference designs as starting points And rapid prototyping Implement design using the appropriate development boards Test the design with actual video signals © 2011 Altera Corporation - Public 31 Next Step Coming Video Framework Workshop, please contact Altera Sales orDistributors for more information Provide an introduction to Video and Image Processing suite including the cores, reference designs, roadmap, video framework, etc. For information on: DDR-based Memory controllers http://www.altera.com/education/training/courses/IMEM210 http://www.altera.com/technology/memory/mem-index.jsp SOPC Builder http://www.altera.com/education/training/courses/OEMB1115 http://www.altera.com/literature/lit-sop.jsp System Console http://www.altera.com/education/training/courses/OEMB1117 http://www.altera.com/literature/ug/ug_system_console.pdf NIOS-II Flow http://www.altera.com/education/training/courses/IEMB115 http://www.altera.com/literature/lit-nio2.jsp Video and Image Processing Suite http://www.altera.com/products/ip/dsp/image_video_processing/m-alt-vipsuite.html http://www.altera.com/end-markets/broadcast/1080p/bro-1080p.html http://www.altera.com/support/refdesigns/sys-sol/broadcast/ref-format-conversion.html On-chip Debugging, including SignalTap-II http://www.altera.com/education/training/courses/ODSW1164 http://www.altera.com/support/software/debugging/sof-qts-debugging.html © 2011 Altera Corporation—Internal 32 Backup © 2011 Altera Corporation—Internal 33