slides of the talk - Vision and Image Processing (VIP) Lab

Introduction to

Deinterlacing

by Mark Korhonen

Copyright © Magnum Semiconductor, Unpublished

Overview:

Introduction To Deinterlacing

Background

• Progressive video

• Interlaced video

• Deinterlacing

Basic Deinterlacing Algorithms

• Weave

• Vertical Interpolation (aka Bob)

Advanced Deinterlacing Algorithms

• Diagonal Interpolation

• Cadence Detection

• Motion Adaptive Deinterlacing (MADI)

• Motion Compensated Deinterlacing (MCDI)

Deinterlacing Applications

2


Background: Progressive Video

A complete video frame is displayed at regular intervals

View on computers and digital televisions

Example resolutions and frame rates:

• SD: 720x480p30, 720x576p25

• HD: 1280x720p60, 1920x1080p30

Sources: film (movies), animation, progressive cameras

Example: 30p = 30 frames/s

3

Picture source: http://lurkertech.com/lg/fields/


Background: Interlaced Video

Frames of video are sampled at two time intervals

• Even rows sampled at: 2t

• Odd rows sampled at: 2t + 1

Terminology:

• Top field: even rows in the frame

• Bottom field: odd rows in the frame

• Field polarity: indicates if a field is a top field or bot. field

Example: 60i = 60 fields/s

4

Picture source: http://lurkertech.com/lg/fields/


Background: Interlaced Video

Why interlaced video exists

• Double frame rate w/ same bandwidth = smoother motion

• Original television formats are all interlaced

– NTSC=720x480i60

– PAL=720x576i50

• 1920x1080i60 exists because

– Processing requirements for 1920x1080p60 are very high

– Motion is smoother than 1920x1080p30

– More detail than 1280x720p60

5


Background: Deinterlacing

Deinterlacing: convert interlaced video to progressive

• Input: X field/s (e.g. i60)

• Output: X frames/s (e.g p60)

• This process generates X “missing” fields/s

6


Background: Deinterlacing

Fields typically used to generate the missing field

• The current field is the opposite polarity of the field that needs to be generated, but displayed at the same time

– E.g. output = bottom field, curr = top field

• The previous and next fields are the same polarity as the field that needs to be generated

– E.g. output = bottom field, prev & next = bottom fields

7


Basic Deinterlacing: Weave

Weave: use “prev” or “next” as the missing field

• For still images and progressive content, no missing data

• Very severe artifacts for moving interlaced content

8

Picture source: doom9.org


Basic Deinterlacing: Weave

Extension

• If stationary, average the previous and next fields together as the missing field

– Noise reduction (averaging two identical images reduces noise)

– Handles fades better if luminance changing every field

Output[x,y] = m prev[x,y] + (1-m) next[x,y]

• m is typically 0.0, 0.5 or 1.0

9


Basic Deinterlacing: Bob

Bob (aka vertical interpolation)

• Generate the opposite polarity by interpolating along vertical columns in the current field

Implementations

• Line doubling – just use current field

– e.g. Output[x,y] = curr[x,y]

• 2-tap FIR filter – average the line above and below

– e.g. Output[x,y] = 0.5 curr[x,y] + 0.5 curr[x,y+1]

• 8-tap FIR filter – better frequency response

10


Basic Deinterlacing: Bob

Bob Video Quality

• Looks good for most video content

– Still content without a lot of detail

– Moving content (hard to see detail on moving objects)

• Interpolation fails if a lot of detail – causes flickering

11


Advanced Deinterlacing:

Diagonal Interpolation

Problem: Bob isn’t ideal for diagonal edges

Solution: Apply FIR filter along a diagonal

12

Picture source: Computer Desktop Encyclopedia



Diagonal Interpolation

Diagonal Edge Detection Algorithm

• Try a bunch of different angles and see which is best

– Pattern recognition problem: classify edges as a supported angle

• Determine which angle is most correlated at each pel

• We can assume that the angle of the edge is wide

» Allows using neighbouring pels to improve accuracy

13



Cadence Detection

Observation: If interlaced video was generated from a progressive source, we can safely weave it with no artifacts

Sample cadences

14

Picture source: http://www.infocellar.com/television




Detection Area

• Size of cadence area can be variable

– e.g. entire field, 16x16 block, every pel

• Detecting entire fields sufficient for ~99% of video

• Smaller areas only needed if video is a mixed source

– moving interlaced text over progressive video

• e.g. weather warnings on a TV movie

– different frame rates of progressive video edited together

• e.g. source of border was 30p, but source of contents were 24p

15




Cadence Algorithm 1: detect regular repeated fields

• For 3:2, every 5th field is a repeat

– SAD will be very low every 5th field  called Inverse Telecine

• Very robust for a lot of content

• Doesn't work for 2:2 (no repeats), or changing cadences

(e.g. slow-motion replays, edited video)

16

Picture source: http://www.infocellar.com/television




Cadence Algorithm 2: detect weaving artifacts

• 1) Weave curr with prev, 2) Weave curr with next

• Determine if weaving artifacts are less in a field

– Pattern recognition problem: classify into three states

• Progressive – weave with previous

• Progressive – weave with next

• Interlaced – perform other deinterlacing

17




Cadence Algorithm 2: detect weaving artifacts

• Tricky for stationary content and at scene changes

– Fairly easy to compensate

• Tricky for vertical motion of detailed video

– e.g. pan of venetian blinds

• Can be tricky for video that looks like it already contains weaving artifacts

– Artifacts introduced by video compression can look like weaving artifacts

– Certain textures

18



Motion Adaptive Deinterlacing (MADI)

Idea: weave stationary areas, interpolate moving areas

Stationary detection = pattern recognition problem

• Size of stationary area can vary (e.g. entire field, 16x16 block, every pel)

– Smaller stationary areas = less flickering but more computation

• Possible implementations:

– SAD of prev and next

• Watch out for periodic motion

– min SAD of last X fields

• Watch out for how long it takes stationary regions to be detected

19



Motion Adaptive Deinterlacing (MADI)

Basic algorithm:

• Inputs:

– Weave – generated from previous and/or next

– Inter – generated from current (diagonal interpolation)

• Output[x,y] = (K) Weave[x,y] + (1-K) Inter[x,y]

– K ~= 1 for stationary/progressive areas (weave)

– K ~= 0 for moving areas (interpolation)

NOTES:

• Fades may need special handling

– e.g. bias more towards interpolation

• x and y advance through regions in the video

– Could be every pel, 16x16 blocks, or the entire field

20



Motion Compensated Deinterlacing (MCDI)

Idea:

• If we can’t weave a particular region, could we weave a motion compensated version of this region from curr, prev or next?

Comments:

• Potential for preserving all of the detail

• Very difficult to do correctly

– Especially rotations, zooms, morphing, lighting changes, etc.

– Objects can get covered/uncovered

– Even small errors can look very bad

• weaving artifacts, ghosting around edges, weird motion, etc.

• Motion compensation should be done for every pel

– Very computationally demanding

21



Motion Compensated Deinterlacing (MCDI)

Basic Algorithm:

• Input fields

– MC_curr[x,y] = curr[x + dx_c, y + dy_c]

– MC_prev[x,y] = prev[x + dx_p, y + dy_p]

– MC_next[x,y] = next[x + dx_n, y + dy_n]

– MADI[x,y] = pel generated using motion adaptive deinterlacing

• Output:

MCDI

 p



MC _ prev

 

 q



MC _ curr x ,

 r



MC _ next

 



( 1

 p

 q

 r )



MADI x ,

• p + q + r ~= 1 if high confidence in motion compensation

• p + q + r ~= 0 if low confidence in motion compensation

– Confidence in the motion compensation can be based on

• The smoothness of the motion vector field

• Estimated SAD (true SAD unknown because the field is missing)

• Consistency of motion from field to field

22



Frame rate conversion from interlaced to interlaced

• No deinterlacing = motion judder

– Need to drop a pair of fields – increases motion judder

• Deinterlacing = smoother motion

– Requires excellent quality deinterlacing

• Deinterlaced fields will be deinterlaced twice

23



Upscaling interlaced video requires deinterlacing

• Perserves more detail

• Minimizes aliasing

24



If the output frame rate is low, flicker can be severe

• If input frame rate is high enough

– In the output, pick a polarity that only use fields from the input

– The other polarity will always be generated (e.g. via interpolation)

• This works because flicker is caused by alternating rows between a) original and b) interpolated – we have removed the alternating

25


Appendices

26


Video Resolutions

Common video resolutions

• SD: 720x480 (NTSC), 720x576 (PAL)

• HD: 1280x720, 1920x1080

• Sub SD: VGA, QVGA, CIF, QCIF, etc.

Definitions

• pels = 1x1 area of video data

• pixel = 1x1 area on video display

• E.g. 720x480 video is 720 pels wide, but may be displayed on a television with 1920 pixels

27


Color Formats

RGB vs. YUV

• Y = greyscale, making deinterlacing decisions primarily based on Y is simpler (but not as accurate)

• Data in U and V isn't as critical so typically downsampled

– 4:2:0 vs 4:2:2 vs 4:4:4

28


slides of the talk - Vision and Image Processing (VIP) Lab

Introduction to

Deinterlacing

by Mark Korhonen

Appendices

Related documents

Products

Support

slides of the talk - Vision and Image Processing (VIP) Lab

Introduction to

Deinterlacing

by Mark Korhonen

Appendices

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib