10-701 and 15-781 Machine

Learning, 2005 (original) (raw)

It is hard to imagine anything more fascinating than systems that automatically improve their own performance through experience. Machine learning deals with computer algorithms for learning from many types of experience, ranging from robots exploring their environments, to mining pre-existing databases, to actively exploring and mining the web. This course is designed to give PhD students a thorough grounding in the methodologies, technologies, mathematics and algorithms needed to do research in learning and data mining, or to apply learning or data mining techniques to a target problem.

The topics of the course draw from classical statistics, from machine learning, from data mining, from Bayesian statistics and from statistical algorithmics.

Students entering the class with a pre-existing working knowledge of probability, statistics and algorithms will be at an advantage, but the class has been designed so that anyone with a strong numerate background can catch up and fully participate.

IF YOU ARE ON THE WAIT LIST:This class if now fully subscribed. You may want to consider the following options:

**Class lectures:**Tuesdays & Thursdays 10:30am-11:50am, Wean Hall 7500 starting on Tuesday September 13th, 2005

Review sessions: Thursdays 5-6pm, Location NSH 1305, starting on thursday September 15. TA's will cover material from lecture and the homeworks, and answer your questions. These review sessions are optional (but very helpful!).

Module

Date

Lecture topic and readings

Lecturer

Homeworks

Optional warm-up

Thu Sep 8

Optional lecture: warm-up review of some basic probability concepts.

Moore

Overview and a Machine Learning algorithm

Tu Sep 13

Machine Learning, Function Approximation, Decision Tree learning

Mitchell

Review of probability,
Maximum likelihood estimation, MAP estimation

Th Sep 15

Fast tour of useful concepts in probability

Moore

HW1
pdf ps.gz Corrections Solutions

Tu Sep 20

MLE and MAP estimation

Moore

Linear models

Th Sep 22

Linear Regression and Basis Functions

Moore

Naive Bayes

Tu Sep 27

Bayesian classifiers, Naive Bayes classifier, MLE and MAP estimates

Mitchell

HW1 due
HW2
pdf train-1.txt test-1.txt plotGauss.m Solutions

Logistic regression

Discriminative and Generative Models

Th Sep 29

Logistic regression, Generative and discriminative classifiers, maximizing conditional data likelihood, MLE and MAP estimates.

Mitchell

Non-linear models
Neural Networks

Tu
Oct 4

Neural networks and gradient descent

Mitchell

Th
Oct 6

Cross-validation and instance-based learning

Moore

HW2 due

Gaussian Mixture Models

Tu
Oct 11

Cross-validation continued

Moore

Th
Oct 13

no lecture

Midterm Exam

(solutions)

Tu
Oct 18

Covers everything up to this date. Open book, notes. Closed computer.

Come to class by 10.30am promptly. You will then have 80 minutes to answer six mostly-short questions on material covered in the lectures and readings up to and including October 11th. We strongly advise you to practice using previous exams, so you know what to expect. try doing the previous exams first, and then look at the solutions. You will be allowed to look at your notes in class, but don't rely on this because you will run out of time unless you are sufficiently familiar with the material that you can just do the questions without needing to look up the techniques.

In addition, to help prepare, there will be a review at the recitation session at 5pm Thursday Oct 13th, and there will be another review on Monday Oct 17th, 6pm-7.30pm in NSH 1305.

Previous examinations for practice.

Project proposals due

Computational learning theory

Th
Oct 20

PAC Learning I: sample complexity, agnostic learning

Mitchell

HW3 ds2.txt

Solution

Tu
Oct 25

PAC Learning II: VC dimension, SRM, Mistake bounds

Mitchell

Margin based approaches

Th
Oct 27

SVMs, kernels, and optimization methods

Moore

RecitationHW3

Graphical Models

Tu
Nov 1

Bayes nets: representation, conditional independence

Mitchell

HW3 due

Th
Nov 3

Bayes nets: inference, variable elimination, etc.

Moore

Recitation

Tu
Nov 8

Bayes nets: learning parameters and structure (fully observed data, and begin EM)

Goldenberg

EM and semi-supervised learning

Th
Nov 10

EM for Bayes networks and Mixtures of Gaussians

Mitchell

HMMs

Tu
Nov 15

Hidden Markov Models: representation and learning

Moore

Time series models

Th
Nov 17

Graphical Models: an overview of more advanced probabistic models that fall under a category called Graphical Models. This lecture defines and talks about specifric instances, such as Kalman filters, undirected graphs and Dynamic Bayesian Networks

Goldenberg

Final project reports due

Mon Nov 21

Project poster session: 4-6:30pm in the Newell-Simon Hall Atrium

Project poster session

Dimensionality reduction

Tu
Nov 22

Dimensionality Reduction: Feature selection, PCA, SVD, ICA, Fisher discriminant

Mitchell

Tu
Nov 29

Advanced topic: Machine Learning and Text Analysis

Mitchell

HW4
missing.csv
EM notes
Inference notes Solutions

Markov models

Th
Dec 1

Markov decision processes: Predicting the results of decisions in an uncertain world.

Moore

Tu
Dec 6

Reinforcement learning: Learning policies to maximize expected future rewards in an uncertain world.

Moore

Th
Dec 8

Scaling: Some of Andrew's favorite data structures and algorithms for tractable statistical machine learning.

Moore

HW4 due

Final Exam

Monday Dec 19

December 19 8:30-11:30a.m at HH B103 and HH B131 (Hammerschlag Hall). No rescheduling possible. open book, open notes, closed computer.

HMM/MDP Review

Dimension Reduction

HMM

Note to people outside CMU: Please feel free to reuse any of these course materials that you find of use in your own courses. We ask that you retain any copyright notices, and include written notice indicating the source of any materials you use.