Computer
Vision
Annie Bashaw, Han Shen
1
The Problem
2
The Problem
• Say you want to make
an embedded system
that tracks and follows a
ball
• Where do we start?
https://i.ytimg.com/vi/3NWtnL5Mmdw/maxresdefault.jpg
3
The Solution
4
What is Computer Vision?
• Computers extract and analyze useful information from a single image or
a series of images
• Try to understand 3D scenes from 2D images
• Basic Goals
•
•
•
Exceed human vision
Make vision methods fast and reliable
Make vision methods work in a variety of environments
http://www.societyofrobots.com/images/programming_co
mputer_vision_servoing.jpg
5
Overview
• 3 Applications
• Hardware Solutions
•
CMUcam
• Software Solutions
•
•
OpenCV
Blob Tracking Example
• Future of Computer Vision
https://upload.wikimedia.org/wikipedia/en/thumb/0/
08/CVoverview2.svg/726px-CVoverview2.svg.png
6
Applications
7
1.Kinect
https://cdn2.hubspot.net/hub/13401/file-13222345-jpg/images/kinect-resized-600.jpg
8
1.Kinect
http://neurogami.com/presentations/KinectForArtists/kinect-images/aT2uupuVXHnJKTsU.large.jpg
9
http://electronicdesign.com/embedded/how-microsoft-s-primesense-based-kinect-really-works
http://i2.ytimg.com/vi/p2nqxyywbQI/0.jpg
10
2. Microsoft Handpose
• Technology to track the
sophisticated and nuanced
hand motions of people
http://tech-review.com/wp-content/uploads/2015/05/microsoft-with-handpose.jpg
• Uses a camera to track a
person’s hand movements
•
Can be used to translate sign
language
•
Potential uses in gaming
http://1u88jj3r4db2x4txp44yqfj1.wpengine.netdna-cdn.com/wpcontent/uploads/2015/04/microsoft_research_handposes.png
11
3. Autonomous Vehicle
• Navigation and detecting obstacles
• Submersibles, land based vehicles, aerial vehicles, unmanned aerial
vehicles
• Space rovers
Source: Nvidia
12
http://www.inquisitr.com/1236826/autonomous-vehicles-are-the-wave-of-the-future-says-stephen-diaco/
13
Back to the Problem...
14
http://vision.mas.ecp.fr/Personnel/teboul/img/vision.jpg
15
Cameras
Gather information from multidimensional data and images
• Cameras are powerful sensors
• Analog or digital image sensors
• CCD sensor versus CMOS image sensor
http://www.2mcctv.com/blog/wp-content/uploads/2012/07/ccd-vs-cmos-image-sensor.jpg
16
CMUcam
Open source programmable embedded color vision sensors
• Real-time vision processing
• Low cost, low power
• Popular for making small, mobile robots
17
CMUcam, continued
• Consists of small video camera
and microcontroller with a
serial interface
• Lightweight interface allows it
to be accessed by
microcontrollers
• On-board microprocessor
supports image processing and
blob tracking
http://i74.photobucket.com/albums/i241/cmucam/Image205_zpsbeb496c2.jpg
Link to specs: http://www.cmucam.org/projects/cmucam5
18
http://www.oocities.org/theprojectstore/camfrontsmaller.jpg
19
Processors
Computer Vision is computationally demanding.
Many applications require:
Processor Types
• High performance
• General-purpose CPUs
• Programmability
• Graphics Processing Units
• Low cost
• Digital Signal Processors
• Energy efficiency
• Field Programmable Gate
Arrays
20
Other Hardware
Considerations
• Cost
• Power requirements/source
• OS support
• User support
• Display system
• Resolution
• Low light
• Pan and tilt
https://www.ecse.rpi.edu/Homepages/qji/CV/3dvision_intro.pdf
• Weight
21
Embedded Vision Algorithms
22
Typical Pipeline
http://electronicdesign.com/sitefiles/electronicdesign.com/files/uploads/2015/02/Synopsys_Thompson_F1.gif
23
Possible Capabilities
•
•
•
•
•
•
•
•
Scene reconstruction
Event detection
Video tracking
Object recognition
Object pose estimation
Learning
Motion estimation
Image restoration
http://crcv.ucf.edu/projects/MP2Tracker/Experiment%20result_3.png
http://www.doc.ic.ac.uk/~rnewcomb/CVPR2010/images/exampleOf3.png
24
Detection/Segmentation
Deciding which image points are
relevant for further processing
• Selecting specific set of
interest points
• Segmentation of one or
multiple image regions
containing a certain object
http://vision.in.tum.de/_media/spezial/bib/nieuwenhuis-et-alijcv13.jpg
25
OpenCV
Library for computer vision
http://static1.squarespace.com/static/56385377e4b0f861c58014
41/t/5661480be4b0766d427cfae5/1449216040970/?format=500
w
• C++, C, Python, Java, and MATLAB interfaces
• Supports Windows, Linux, Android, and Mac OS
• Provides more than 2500 optimized algorithms
26
http://www.embedded-vision.com/sites/default/files/articleimages/opencv_diagram.png
27
Image Segmentation Using
OpenCV Watershed Algorithm
• View images as topographic
• High intensity denotes while low intensity denotes valleys
• Fill isolated valleys with different colored “water” (labels)
• As the water rises, colors will merge
• Avoid this by building barriers
• This will give a segmented result
28
Application of Watershed Algorithm
• Lane segmentation
• Obstacles detection
• Extract information from pictures
• Function call: cv::watershed(InputArray image, InputOutputArray markers)
http://cmm.ensmp.fr/~beucher/prometheus.html
29
Blob Tracking
• What is a Blob?
•
A group of connected pixels in an image that share some common property
•
The goal of blob detection is to identify and mark the regions of blobs
• How does blob detection work
1. Thresholding
2. Grouping
3. Merging
4. Center & Radius Calculation
http://www.learnopencv.com/blob-detection-using-opencv-python-c/
30
Blob Tracking
• Filtering blobs by
•
Color
•
Size
•
Shape
http://i.stack.imgur.com/o3PU9.jpg
31
32
33
Putting It All Together...
34
https://i.ytimg.com/vi/3BJFxnap0AI/maxresdefault.jpg
35
Swimming Pool Safety
• From 2005 - 2009, there were an
average of 3,533 fatal drownings
• ~20% of drownings occur in
public pools with present
lifeguards
• The Poseidon system from MG
International
http://www.embedded-vision.com/what-is-embedded-vision/case-studies
36
The Future
37
The Future of Computer Vision
• Mark Zuckerberg’s bold
predictions: is he far off?
• Keyword searches versus
image and video searches
• The future is wearable
http://technology.inquirer.net/files/2013/12/computer-blind.jpg
38
Summary
• Computer vision will allow
your embedded system to
comprehend complex
images
• Cameras are just really
powerful sensors
•
Solution: CMUcam
• Algorithms are not easy, but
there’s a tool for that
•
http://vislab.isr.ist.utl.pt/wp-content/uploads/2012/12/creationresized.jpg
Solution: OpenCV
39
Questions?
40
Resources
http://www.bmva.org/visionoverview
http://www.cdc.gov/HomeandRecreationalSafety/Water-Safety/waterinjuries-factsheet.html
https://en.wikipedia.org/wiki/Computer_vision
https://cseweb.ucsd.edu/classes/wi04/cse291-c/lec1.pdf
http://opencv.org/about.html
http://www.cmucam.org/
http://mil.ufl.edu/5666/classes/Computer_Vision_Presentation.pdf
http://www.computervisiononline.com/blog/robot-always-wins-rock-paper-scissors
http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/how-google-self-driving-car-works
http://www.theatlantic.com/technology/archive/2012/09/driverless-cars-would-reshape-automobiles-and-the-transit-system/262953/
http://www.tkhgroup.com/en/about-tkh-group/technologies-and-vertical-growth-markets/technologies-and-vertical-growth-markets
http://docs.opencv.org/3.1.0/d3/db4/tutorial_py_watershed.html#gsc.tab=0
http://linuxgizmos.com/nvidia-aims-octa-core-256-gpu-tegra-x1-soc-at-smart-cars/http://users.dickinson.edu/~jmac/selectedtalks/kinect.pdf
http://www.robots.ox.ac.uk/~nema/publications/Fletcher01.pdf
http://www.cs.cornell.edu/courses/cs7670/2011fa/lectures/zhaoyin_kinect.pdf
https://www.ecse.rpi.edu/Homepages/qji/CV/3dvision_intro.pdf
http://www.businessinsider.com/mark-zuckerbergs-prediction-about-computer-vision-2015-7
http://www.bdti.com/private/pubs/BDTI_ESC_Embedded_Vision.pdf
http://openeuroscience.com/software/computer-vision-and-motion-tracking-software/
http://www.embedded-vision.com/
http://www.computervisionblog.com/2015/01/from-feature-descriptors-to-deep.html
41
Appendix
42
More Applications
43
Janken (Rock-Paper-Scissors) Robot
• Ishikawa Watanabe, University
of Tokyo, 2012
• Detects human hand motions
with high speed vision
• Performs at 1 ms
http://www.k2.t.u-tokyo.ac.jp/fusion/Janken/index-e.html
44
Industrial Machine Vision
•
•
•
•
Info is extracted to support a
manufacturing process
Quality control
Used for measurement of the
position and orientation of
something for a robot arm to
pick up
Used in agriculture to remove
undesirable food (optical
sorting)
Source: TKH Group
45
Medical Image Analysis
•
•
•
•
Used for making medical diagnoses
of a patient
In the form of microscopy images,
X-ray images, angiography images,
ultrasonic images, and tomography
images
Data can be enhanced so that it can
be interpreted by humans
Used with forensics and biometrics
https://www.cbica.upenn.edu/sbia/img/sbia-collage.jpg
46
More Kinect Details
47
Kinect
• Inferring body position
•
•
Compute a depth map - using structured light
• The depth map is constructed by analyzing a speckle pattern of infrared laser light
• Uses an infrared projector and sensor
• Depth from focus and stereo
Infer body position
• Body parts are inferred using a randomized decision forest
• Learned from over 1 million training examples
• Probability distribution
48
Infrared laser light with a speckle pattern
https://azttm.files.wordpress.com/2011/04/kinect-pattern_2.png
49
Depth from stereo using parallax
http://docplayer.net/docs-images/17/102893/images/17-0.jpg
50
Watershed Algorithm Walkthrough
51
Watershed Algorithm
Find approximate estimate
of object location
The following several images are taken from
http://docs.opencv.org/3.1.0/d3/db4/tutorial_py_watershed.
html#gsc.tab=0
52
Watershed Algorithm
• Remove any small white noises
• Use erosion to find coins
• Use dilation to find background
53
Watershed Algorithm
Boundaries can be found from subtracting the sure foreground from the sure
background
54
Watershed Algorithm
Now we know what regions in the image are for sure coins
55
Watershed Algorithm
Mark image with different
colored “water”
56
Watershed Algorithm
“Fill up” image with water until colors merge; this gives image boundaries
57