Important Announcement
• Lab this week
– Not in the usual room
– Will meet in Friend 017
1
Digital Cameras
Engineering Math Physics (EMP)
Jennifer Rexford
http://www.cs.princeton.edu/~jrex
2
Image Transmission Over Wireless Networks
• Wireless networks
– Wireless technology
– Acoustic waves and electrical signals
– Radios
• Image capture and compression
– Inner-workings of a digital camera
– Manipulating & transforming a matrix of pixels
– Implementing a variant of JPEG compression
• Video over wireless networks
– Video compression and quality
– Transmitting video over wireless
– Controlling a car over a radio link
3
Traditional Photography
• A chemical process, little
changed from 1826
• Taken in France on a
pewter plate
• … with 8-hour exposure
The world's first photograph
4
Image Formation
Digital Camera
Film
Eye
5
Image Formation in a Pinhole Camera
• Light enters a darkened chamber through pinhole
opening and forms an image on the further surface
6
Aperture
• Hole or opening where light enters
–Or, the diameter of that hole or opening
• Pupil of the human eye
–Bright light: 1.5 mm diameter
–Average light: 3-4 mm diameter
–Dim light: 8 mm diameter
• Camera
–Wider aperture admits more light
–Though leads to blurriness in the
objects away from point of focus
7
Shutter Speed
• Time for light to enter camera
–Longer times lead to more light
–… though blurs moving subjects
• Exposure
–Total light entering
the camera
–Depends on aperture
and shutter speed
8
Digital Photography
• Digital photography is an
electronic process
• Only widely available in the
last ten years
• Digital cameras now surpass
film cameras in sales
• Polaroid closed its film plants…
9
Image Formation in a Digital Camera
+10V
Photon
++++++







+
A sensor converts one
kind of energy to another
• Array of sensors
– Light-sensitive diodes convert photons to electrons
– Buckets that collect charge in proportion to light
– Each bucket corresponds to a picture element (pixel)
10
CCD: Charge Coupled Device
CCD sensor
• Common sensor array used in digital cameras
– Each capacitor accumulates charge in response to light
• Responds to about 70% of the incident light
– In contrast, photographic film captures only about 2%
• Also widely used in astronomy telescopes
11
Sensor Array: Image Sampling
Pixel (Picture Element): single point in a graphic image
12
Sensor Array: Reading Out the Pixels
• Transfer the charge from
one row to the next
• Transfer charge in the serial
register one cell at a time
• Perform digital to analog
conversion one cell at a time
• Store digital representation
Digital-to-analog
conversion
13
Sensor Array: Reading Out the Pixels
14
More Pixels Mean More Detail
1600 x 1400
1280 x 960
640 x 480
15
The
2272 x 1704
hand
The
320 x 240
hand
16
Representing Color
• Light receptors in the human eye
– Rods: sensitive in low light, mostly at periphery of eye
– Cones: only at higher light levels, provide color vision
– Different types of cones for red, green, and blue
• RGB color model
– A color is some combination of red, green, and blue
– Intensity value for each color
 0 for no intensity
 1 for high intensity
– Examples
 Red: 1, 0, 0
 Green: 0, 1, 0
 Yellow: 1, 1, 0
17
Representing Image as a 3D Matrix
• In the lab this week…
– Matlab experiments with digital images
• Matrix storing color intensities per pixel
– Row: from top to bottom
– Column: from left to right
– Color: red, green, blue
• Examples
1
2
3
1 2
– M(3,2,1): third row, second column, red intensity
– M(4,3,2): fourth row, third column, green intensity
18
Limited Granularity of Color
• Three intensities, one per color
– Any value between 0 and 1
• Storing all possible values take a lot of bits
– E.g., storing 0.368491029692069439604504560106
– Can a person really differentiate from 0.36849?
• Limiting the number of intensity settings
– Eight bits for each color
– From 00000000 to 11111111
– With 28 = 256 values
• Leading to 24 bits per pixel
 Red: 255, 0, 0
 Green: 0, 255, 0
 Yellow: 255, 255, 0
19
Number of Bits Per Pixel
• Number of bits per pixel
– More bits can represent a wider range of colors
– 24 bits can capture 224 = 16,777,216 colors
– Most humans can distinguish around 10 million colors
8 bits / pixel / color
4 bits / pixel / color
20
Separate Sensors Per Color
• Expensive cameras
– A prism to split the light into three colors
– Three CCD arrays, one per RGB color
21
Practical Color Sensing: Bayer Grid
• Place a small color filter
over each sensor
• Each cell captures
intensity of a single color
• More green pixels, since
human eye is better at
resolving green
22
Practical Color Sensing: Interpolating
• Challenge: inferring what we can’t see
– Estimating pixels we do not know
• Solution: estimate based on neighboring pixels
– E.g., red for non-red cell averaged from red neighbors
– E.g., blue for non-blue cell averaged from blue neighbors
Estimate “R” and “B” at
the “G” cells from
neighboring values
23
Interpolation
• Examples of interpolation
and
makes
and
makes
and
makes
• Accuracy of interpolation
– Good in low-contrast areas (neighbors mostly the same)
– Poor with sharp edges (e.g., text)
24
Are More Pixels Always Better?
• Generally more is better
– Better resolution of the picture
– Though at some point humans can’t tell the difference
• But, other factors matter as well
– Sensor size
– Lens quality
– Whether Bayer grid is used
• Problem with too many pixels
– Very small sensors catch fewer photons
– Much higher signal-to-noise ratio
• Plus, more pixels means more storage…
25
Digital Images Require a Lot of Storage
• Three dimensional object
– Width (e.g., 640 pixels)
– Height (e.g., 480 pixels)
– Bits per pixel (e.g., 24-bit color)
• Storage is the product
– Pixel width * pixel height * bits/pixel
– Divided by 8 to convert from bits to bytes
• Example sizes
– 640 x 480: 1 Megabyte
– 800 x 600: 1.5 Megabytes
– 1600 x 1200: 6 Megabytes
26
Compression
• Benefits of reducing the size
– Consume less storage space and network bandwidth
– Reduce the time to load, store, and transmit the image
• Redundancy in the image
– Neighboring pixels often the
same, or at least similar
– E.g., the blue sky
• Human perception factors
– Human eye is not sensitive
to high frequencies
27
Compression Pipeline
• Sender and receiver must agree
– Sender/writer compresses the raw data
– Receiver/reader un-compresses the compressed data
compress
uncompress
• Example: digital photography
compress
uncompress
28
Two Kinds of Compression
• Lossless
– Only exploits redundancy in the data
– So, the data can be reconstructed exactly
– Necessary for most text documents (e.g., legal
documents, computer programs, and books)
• Lossy
– Exploits both data redundancy and human perception
– So, some of the information is lost forever
– Acceptable for digital audio, images, and video
29
Lossless: Huffman Encoding
• Normal encoding of text
– Fixed number of bits for each character
• ASCII with seven bits for each character
– Allows representation of 27=128 characters
– Use 97 for ‘a’, 98 for ‘b’, …, 122 for ‘z’
• But, some characters occur more often than others
– Letter ‘a’ occurs much more often than ‘x’
• Idea: assign fewer bits to more-popular symbols
– Encode ‘a’ as “000”
– Encode ‘x’ as “11010111”
30
Lossless: Huffman Encoding
• Challenge: generating an efficient encoding
– Smaller codes for popular characters
– Longer codes for unpopular characters
English Text: frequency distribution
Morse code
31
Lossless: Run-Length Encoding
• Sometimes the same symbol repeats
– Such as “eeeeeee” or “eeeeetnnnnnn”
– That is, a run of “e” symbols or a run of “n” symbols
• Idea: capture the symbol only once
– Count the number of times the symbol occurs
– Record the symbol and the number of occurrences
• Examples
– So, “eeeeeee” becomes “@e7”
– So, “eeeeetnnnnnn” becomes “@e5t@n6”
• Useful for fax machines
– Lots of white, separate by occasional black
32
Joint Photographic Experts Group
• Starts with an array of pixels in RGB format
– With one number per pixel for each of the three colors
– And outputs a smaller file with some loss in quality
• Exploits both redundancy and human perception
– Transforms data to identify parts that humans notice less
– More about transforming the data in Wednesday’s class
Uncompressed: 167 KB
Good quality: 46 KB
Poor quality: 9 KB33
Joint Photographic Experts Group (JPEG)
108 KB
8 KB
34 KB
Lossy
compression
34 34
New Era of Computational Photography
• Beyond manual editing of photographs
– E.g., to crop, lighten/darken, sharpen edges, etc.
• Post-processing of a single photography
– De-blur photos marred by camera shake
– Changing the depth of field or vantage point
• Combining multiple pictures
– Creating three-dimensional representations
– Stitching together photos for a larger view
– Creating a higher-resolution picture of a single scene
• <Insert your idea here>
http://www.news.com/8301-13580_3-9882019-39.html
35
Conclusion
• Conversion of information
– Light (photons) and a optical lens
– Charge (electrons) and electronic devices
– Bits (0s and 1s) and a digital computer
• Combines many disciplines
– Physics: lenses and light
– Electrical engineering: charge coupled device
– Computer science: manipulating digital representations
– Mathematics: compression algorithms
– Psychology/biology: human perception
• Next class: compression algorithms
36
Important Announcement
• Lab this week
– Not in the usual room
– Will meet in Friend 017
37