Lecture 1: Aug. 20, 2012
EE645 Machine Learning
Fall, 2012
•  Instructor: Tony Kuh
•  POST 205E / 484 Holmes
•  Dept. of Electrical Engineering
•  University of Hawaii
•  Phone: 956-7527, 956-4214
•  Email: kuh@hawaii.edu
Department of
Electrical
Engineering
1
Lecture 1: Aug. 20, 2012
Preliminaries
•  Class Meeting Time: MWF 11:30-12:20 (248
Holmes), after this week twice a week
•  Website: go to my website to get access to EE645
site
•  Office Hours: MW 10-11:30 or by appointment
•  Prerequisites
–  Probability: EE342 or equivalent
•  Random variables, Expectation, Bayes analysis,
Gaussian RVs and Gaussian processes
–  Linear Algebra: vector and matrix operations
–  Programming: Matlab or C experience
Department of
Electrical
Engineering
2
Lecture 1: Aug. 20, 2012
Objectives and Grading
Topics: Machine learning, pattern recognition, signal
processing, neural networks, applications
Objectives: obtain basic understanding and
knowledge of fundamental concepts of machine
learning, learn about current research in area,
conduct project on topic of current research
Grading:
•  Homework: 30%
•  Exam:30%
•  Final project: 40% (oral presentation and written
report)
Department of
Electrical
Engineering
3
Lecture 1: Aug. 20, 2012
Motivation
•  Handling data; process, analyze, and learn from
data
•  Processing data: CPUs and storage device
technology have improved dramatically, algorithm
development to process data has not increased
nearly as rapidly
•  Learn from data. Develop paradigms for learning
that mimic features of natural learning for
applications in engineering and science
•  Multidisciplinary area requiring tools from EE, CS,
Statistics, Physics, Math, Biology
Department of
Electrical
Engineering
4
Lecture 1: Aug. 20, 2012
Overview of Course Material
•  Linear algorithms for classification and regression
–  Classification
•  Linear Threshold Unit (Perceptron Learning
Algorithm)
•  Optimum margin classifiers
•  Linear logistic regression
–  Regression (linear unit)
•  LMS algorithm
•  Least squares algorithm
Department of
Electrical
Engineering
5
Lecture 1: Aug. 20, 2012
Overview continued
•  Kernel Methods
–
–
–
–
–
Optimization methods
Kernels
Support Vector Machines
Least Squares kernel algorithms
On-line algorithms
•  More learning algorithms
–
–
–
–
Generative classifier: Naive Bayes
Multilayer networks: Backpropagation
Ensemble learning (boosting, AdaBoost)
Mixture models, EM algorithm
Department of
Electrical
Engineering
6
Lecture 1: Aug. 20, 2012
Overview continued
•  Learning Theory
–  Learning and generalization
–  Structural risk minimization
–  Dimensionality and generalization bounds
•  Unsupervised Learning
–  Component Analysis: PCA, Kernel PCA, ICA
–  Competitive Learning
•  Self – Organizing Feature Maps
•  Vector quantization
•  Reinforcement learning
–  Markov decision processes and dynamic programming
–  TD learning, Q learning
Department of
Electrical
Engineering
7
Lecture 1: Aug. 20, 2012
Historical notes
•  1940s: Hebb, The organization of behavior,
McCulloch-Pitts model, Von Neumann
•  1950s-1960s: Rosenblatt, Minsky-Papert,
Perceptrons, artificial intelligence, Widrow
•  1970s-1980s: Pioneers (Grossberg, Amari,
Kohonen), Hopfield, PDP Group
•  1990s-2000s: Multidisciplinary area (machine
learning, statistics, physics, biology),
mathematical rigor (learning theory, kernel
methods, reinforcement learning,Bayesian
learning, unsupervised learning)
•  2010s: renewed interest “big data”
Department of
Electrical
Engineering
8
Lecture 1: Aug. 20, 2012
Applications
•
•
•
•
Character recognition
Text classification
Biomedical classification: disease diagnosis
Bioinformatics: gene sequencing and protein
classification
•  Time series prediction
•  Social network applications: Netflix, internet
advertising
•  Energy and smart grid applications
Department of
Electrical
Engineering
9
Lecture 1: Aug. 20, 2012
References
•
•
•
•
•
•
•
•
•
•
T. Hastie, R. Tibshirani, and J. Friedman. Elements of Statistical Learning
Data Mining, Inference, and Prediction 2nd ed., Springer, 2009.
Y. S. Abu-Mostafa, M. Magdon-Ismail, and H.-T. Lin, Learning from Data,
AMLBook, 2012.
C. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
S. Haykin. Neural Networks and Learning Machines. 3rd Ed. Prentice Hall,
Englewood Cliffs,NJ, 2008.
R. Duda, P. Hart, and D. Stork. Pattern Classification. 2nd Ed. Wiley, 2000.
J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis.
Cambridge University Press, 2004.
B. Scholkopf and A. Smola. Learning with Kernels: Support Vector
Machines, Regularization, Optimization and Beyond. MIT Press, Cambridge,
MA, 2002.
N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector
Machines: and other Kernel Based Learning Methods. Cambridge University
Press, Cambridge, UK, 2000.
D. Koller and N. Friedman. Probabilistic Graphical Models Principles and
Techniques. MIT Press, 2009.
R. Sutton and A. Barto. Reinforcement Learning: An Introduction (Adaptive
computation and machine learning). Bradford Book. 1998.
Department of
Electrical
Engineering
10
Lecture 1: Aug. 20, 2012
Web resources
•  Organizations
–  IEEE Computational Intelligence Society
–  International Neural Network Society
•  Journals and Conferences
–
–
–
–
–
IEEE Trans. on Neural Networks and Learning Systems
Neural Computation
Neural Networks
Neural Information Processing Systems (NIPS), tutorials
WCCI, IJCNN
•  Online courses (iTunes U)
–  Machine Learning Course: Y. Abu Mostafa, Caltech
–  Machine Learning: A. Ng, Stanford
Department of
Electrical
Engineering
11
Lecture 1: Aug. 20, 2012
Web resource continued
•  Other sites
–  Kernel methods
–  Tutorials (from conferences, short courses)
–  Datasets (e.g. UC Irvine machine learning repository)
Department of
Electrical
Engineering
12