Concept Learning in Machine Learning: Find-S Algorithm

Concept Learning Learning Conjunctive Concepts Inductive Bias References Introduction to Machine Learning CSE474/574: Concept Learning Varun Chandola <chandola@buffalo.edu> Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Outline 1 Concept Learning Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space 2 Learning Conjunctive Concepts Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm 3 Inductive Bias Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Outline 1 Concept Learning Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space 2 Learning Conjunctive Concepts Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm 3 Inductive Bias Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Concept Learning Infer a boolean-valued function c : x → {true,false} Input: Attributes for input x Output: true if input belongs to concept, else false Go from specific to general (Inductive Learning). Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Finding Malignant Tumors from MRI Scans Attributes 1 Shape circular,oval 2 Size large,small 3 Color light,dark 4 Surface smooth,irregular 5 Thickness thin,thick Concept Malignant tumor. Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Malignant vs. Benign Tumor Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Malignant vs. Benign Tumor Malicious Malicious Benign Malicious A large irregularly shaped dark reddish blob. Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Malignant vs. Benign Tumor Malicious Malicious Benign Malicious A large irregularly shaped dark reddish blob. Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Malignant vs. Benign Tumor Malicious Malicious Benign Malicious A large irregularly shaped dark reddish blob. Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Notation X - Set of all possible instances. What is |X |? Example: {circular,small,dark,smooth,thin} D - Training data set. D = {hx, c(x)i : x ∈ X , c(x) ∈ {0, 1}} Typically, |D| |X | Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Representing a Concept - Hypothesis A conjunction over a subset of attributes A malignant tumor is: circular and dark and thick {circular,?,dark,?,thick} Target concept c is unknown Value of c over the training examples is known Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Approximating Target Concept Through Hypothesis Hypothesis: a potential concept Example: {circular,?,?,?,?} Hypothesis Space (H): Set of all hypotheses What is |H|? Special hypotheses: Accept everything, {?,?,?,?,?} Accept nothing, {∅, ∅, ∅, ∅, ∅} Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space Approximating Target Concept Through Hypothesis Hypothesis: a potential concept Example: {circular,?,?,?,?} Hypothesis Space (H): Set of all hypotheses What is |H|? Special hypotheses: Accept everything, {?,?,?,?,?} Accept nothing, {∅, ∅, ∅, ∅, ∅} Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Outline 1 Concept Learning Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space 2 Learning Conjunctive Concepts Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm 3 Inductive Bias Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm A Simple Algorithm (Find-S [2, Ch. 2]) 1 Start with h = ∅ 2 Use next input {x, c(x)} 3 If c(x) = 0, goto step 2 h ← h ∧ x (pairwise-and) 4 5 If more examples: Goto step 2 6 Stop Varun Chandola Pairwise-and rules:    ax :  ax : ah ∧ ax = ? :    ? : Introduction to Machine Learning if if if if ah ah ah ah =∅ = ax 6= ax =? Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Simple Example Target concept {?,large,?,?,thick} How many positive examples can there be? What is the minimum number of examples need to be seen to learn the concept? 1 2 {circular,large,light,smooth,thick}, malignant {oval,large,dark,irregular,thick}, malignant Maximum? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Simple Example Target concept {?,large,?,?,thick} How many positive examples can there be? What is the minimum number of examples need to be seen to learn the concept? 1 2 {circular,large,light,smooth,thick}, malignant {oval,large,dark,irregular,thick}, malignant Maximum? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Simple Example Target concept {?,large,?,?,thick} How many positive examples can there be? What is the minimum number of examples need to be seen to learn the concept? 1 2 {circular,large,light,smooth,thick}, malignant {oval,large,dark,irregular,thick}, malignant Maximum? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Partial Training Data Target concept {?,large,?,?,thick} 1 {circular,large,light,smooth,thick}, malignant 2 {circular,large,light,irregular,thick}, malignant 3 {oval,large,dark,smooth,thin}, benign 4 {oval,large,light,irregular,thick}, malignant 5 {circular,small,light,smooth,thick}, benign Concept learnt: {?,large,light,?,thick} What mistake can this “concept” make? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Partial Training Data Target concept {?,large,?,?,thick} 1 {circular,large,light,smooth,thick}, malignant 2 {circular,large,light,irregular,thick}, malignant 3 {oval,large,dark,smooth,thin}, benign 4 {oval,large,light,irregular,thick}, malignant 5 {circular,small,light,smooth,thick}, benign Concept learnt: {?,large,light,?,thick} What mistake can this “concept” make? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Partial Training Data Target concept {?,large,?,?,thick} 1 {circular,large,light,smooth,thick}, malignant 2 {circular,large,light,irregular,thick}, malignant 3 {oval,large,dark,smooth,thin}, benign 4 {oval,large,light,irregular,thick}, malignant 5 {circular,small,light,smooth,thick}, benign Concept learnt: {?,large,light,?,thick} What mistake can this “concept” make? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Recap of Find-S Objective: Find maximally specific hypothesis Admit all positive examples and nothing more Hypothesis never becomes any more specific Questions Does it converge to the target concept? Is the most specific hypothesis the best? Robustness to errors Choosing best among potentially many maximally specific hypotheses Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Recap of Find-S Objective: Find maximally specific hypothesis Admit all positive examples and nothing more Hypothesis never becomes any more specific Questions Does it converge to the target concept? Is the most specific hypothesis the best? Robustness to errors Choosing best among potentially many maximally specific hypotheses Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Version Spaces 1 {circular,large,light,smooth,thick}, malignant 2 {circular,large,light,irregular,thick}, malignant 3 {oval,large,dark,smooth,thin}, benign 4 {oval,large,light,irregular,thick}, malignant 5 {circular,small,light,smooth,thin}, benign Hypothesis chosen by Find-S: {?,large,light,?,thick} Other possibilities that are consistent with the training data? What is consistency? Version space: Set of all consistent hypotheses. Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Version Spaces 1 {circular,large,light,smooth,thick}, malignant 2 {circular,large,light,irregular,thick}, malignant 3 {oval,large,dark,smooth,thin}, benign 4 {oval,large,light,irregular,thick}, malignant 5 {circular,small,light,smooth,thin}, benign Hypothesis chosen by Find-S: {?,large,light,?,thick} Other possibilities that are consistent with the training data? What is consistency? Version space: Set of all consistent hypotheses. Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm List Then Eliminate 1 2 VS ← H For Each hx, c(x)i ∈ D: Remove every hypothesis h from VS such that h(x) 6= c(x) 3 Return VS Issues? How many hypotheses are removed at every instance? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm List Then Eliminate 1 2 VS ← H For Each hx, c(x)i ∈ D: Remove every hypothesis h from VS such that h(x) 6= c(x) 3 Return VS Issues? How many hypotheses are removed at every instance? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Compressing Version Space More General Than Relationship hj ≥g hk if hk (x) = 1 ⇒ hj (x) = 1 hj >g hk if (hj ≥g hk ) ∧ (hk g hj ) In a version space, there are: 1 2 Maximally general hypotheses Maximally specific hypotheses Boundaries of the version space Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Example 1 {circular,large,light,smooth,thick}, malignant 2 {circular,large,light,irregular,thick}, malignant {?,large,light,?,thick} 3 {oval,large,dark,smooth,thin}, benign {?,large,?,?,thick} 4 {oval,large,light,irregular,thick}, malignant {?,large,light,?,?} 5 {circular,small,light,smooth,thick}, benign Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Example (2) Specific General Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Example (2) Specific {?,large,light,?,thick} General {?,large,?,?,thick} Varun Chandola {?,large,light,?,?} Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Boundaries are Enough to Capture Version Space Version Space Representation Theorem Every hypothesis h in the version space is contained within at least one pair of hypothesis, g and s, such that g ∈ G and s ∈ S, i.e.,: g ≥g h ≥g s Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Candidate Elimination Algorithm 1 2 Initialize S0 = {∅}, G0 = {?, ?, . . . , ?} For every training example, d = hx, c(x)i c(x) = −ve c(x) = +ve 1 Remove from G any g for which g (x) 6= +ve 1 Remove from S any s for which s(x) 6= −ve 2 For every s ∈ S such that s(x) 6= +ve: 2 For every g ∈ G such that g (x) 6= −ve: 1 2 Remove s from S For every minimal generalization, s 0 of s 1 2 If g 0 (x) = −ve and there exists s 0 ∈ S such that g 0 >g s 0 Add g 0 to G If s 0 (x) = +ve and there exists g 0 ∈ G such that g 0 >g s 0 Add s 0 to S 3 Remove from S all hypotheses that are more general than another hypothesis in S Varun Chandola Remove g from G For every minimal specialization, g 0 of g 3 Remove from G all hypotheses that are more specific than another hypothesis in G Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Candidate Elimination Algorithm 1 2 Initialize S0 = {∅}, G0 = {?, ?, . . . , ?} For every training example, d = hx, c(x)i c(x) = −ve c(x) = +ve 1 Remove from G any g for which g (x) 6= +ve 1 Remove from S any s for which s(x) 6= −ve 2 For every s ∈ S such that s(x) 6= +ve: 2 For every g ∈ G such that g (x) 6= −ve: 1 2 Remove s from S For every minimal generalization, s 0 of s 1 2 If g 0 (x) = −ve and there exists s 0 ∈ S such that g 0 >g s 0 Add g 0 to G If s 0 (x) = +ve and there exists g 0 ∈ G such that g 0 >g s 0 Add s 0 to S 3 Remove from S all hypotheses that are more general than another hypothesis in S Varun Chandola Remove g from G For every minimal specialization, g 0 of g 3 Remove from G all hypotheses that are more specific than another hypothesis in G Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Candidate Elimination Algorithm 1 2 Initialize S0 = {∅}, G0 = {?, ?, . . . , ?} For every training example, d = hx, c(x)i c(x) = −ve c(x) = +ve 1 Remove from G any g for which g (x) 6= +ve 1 Remove from S any s for which s(x) 6= −ve 2 For every s ∈ S such that s(x) 6= +ve: 2 For every g ∈ G such that g (x) 6= −ve: 1 2 Remove s from S For every minimal generalization, s 0 of s 1 2 If g 0 (x) = −ve and there exists s 0 ∈ S such that g 0 >g s 0 Add g 0 to G If s 0 (x) = +ve and there exists g 0 ∈ G such that g 0 >g s 0 Add s 0 to S 3 Remove from S all hypotheses that are more general than another hypothesis in S Varun Chandola Remove g from G For every minimal specialization, g 0 of g 3 Remove from G all hypotheses that are more specific than another hypothesis in G Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Example Specific S0 {∅} General G0 {?,?,?,?,?} Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Example Specific S0 {∅} S1 {ci,la,li,sh,th} h{ci,la,li,sh,th}, +vei General G0 {?,?,?,?,?} Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Example Specific S0 {∅} S1 {ci,la,li,sh,th} h{ci,la,li,sh,th}, +vei h{ci,la,li,ir,th}, +vei S2 {ci,la,li,?,th} General G0 {?,?,?,?,?} Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Example Specific S0 {∅} S1 {ci,la,li,sh,th} h{ci,la,li,sh,th}, +vei h{ci,la,li,ir,th}, +vei S2 {ci,la,li,?,th} h{ov,sm,li,sh,tn}, -vei G3 {ci,?,?,?,?},{?,la,?,?,?},{?,?,dk,?,?},{?,?,?,ir,?},{?,?,?,?,th} General G0 {?,?,?,?,?} Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Example Specific S0 {∅} S1 {ci,la,li,sh,th} h{ci,la,li,sh,th}, +vei h{ci,la,li,ir,th}, +vei S2 {ci,la,li,?,th} h{ov,sm,li,sh,tn}, -vei h{ov,la,li,ir,th}, +vei S4 {?,la,li,?,th} G3 {ci,?,?,?,?},{?,la,?,?,?},{?,?,?,?,th} G3 {ci,?,?,?,?},{?,la,?,?,?},{?,?,dk,?,?},{?,?,?,ir,?},{?,?,?,?,th} General G0 {?,?,?,?,?} Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Understanding Candidate Elimination S and G boundaries move towards each other Will it converge? 1 2 3 No errors in training examples Sufficient training data The target concept is in H Why is it better than Find-S? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Not Sufficient Training Examples Use boundary sets S and G to make predictions on a new instance x∗ Case 1: x ∗ is consistent with every hypothesis in S Case 2: x ∗ is inconsistent with every hypothesis in G Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Partially Learnt Concepts - Example {?,la,li,?,th} {?,la,?,?,?} Varun Chandola {?,?,?,?,th} Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Partially Learnt Concepts - Example {?,la,li,?,th} {?,la,li,?,?} {?,la,?,?,th} {?,la,?,?,?} Varun Chandola {?,?,li,?,th} {?,?,?,?,th} Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Partially Learnt Concepts - Example {?,la,li,?,th} {?,la,li,?,?} {?,la,?,?,th} {?,la,?,?,?} {?,?,li,?,th} {?,?,?,?,th} {ci,la,li,sh,th}, ? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Partially Learnt Concepts - Example {?,la,li,?,th} {?,la,li,?,?} {?,la,?,?,th} {?,la,?,?,?} {?,?,li,?,th} {?,?,?,?,th} {ov,sm,li,ir,tn}, ? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Partially Learnt Concepts - Example {?,la,li,?,th} {?,la,li,?,?} {?,la,?,?,th} {?,la,?,?,?} {?,?,li,?,th} {?,?,?,?,th} {ov,la,dk,ir,th}, ? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Partially Learnt Concepts - Example {?,la,li,?,th} {?,la,li,?,?} {?,la,?,?,th} {?,la,?,?,?} {?,?,li,?,th} {?,?,?,?,th} {ci,la,li,ir,tn}, ? Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm Using Partial Version Spaces Halving Algorithm Predict using the majority of concepts in the version space Randomized Halving Algorithm [1] Predict using a randomly selected member of the version space Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Outline 1 Concept Learning Example – Finding Malignant Tumors Notation Representing a Possible Concept - Hypothesis Hypothesis Space 2 Learning Conjunctive Concepts Find-S Algorithm Version Spaces LIST-THEN-ELIMINATE Algorithm Compressing Version Space Analyzing Candidate Elimination Algorithm 3 Inductive Bias Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References How many target concepts can there be? Target concept labels examples in X 2|X | possibilities (C) Qd |X | = i=1 ni Conjunctive hypothesis space H has Qd i=1 ni + 1 possibilities Why is this difference? Hypothesis Assumption Target concept is conjunctive. Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References How many target concepts can there be? Target concept labels examples in X 2|X | possibilities (C) Qd |X | = i=1 ni Conjunctive hypothesis space H has Qd i=1 ni + 1 possibilities Why is this difference? Hypothesis Assumption Target concept is conjunctive. Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Inductive Bias C H Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Inductive Bias {ci,?,?,?,?,?} C H Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Inductive Bias {ci,?,?,?,?,?} ∨ {?,?,?,?,?,th} {ci,?,?,?,?,?} C H Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Bias Free Learning – C ≡ H Simple tumor example: 2 attributes - size (sm/lg) and shape (ov/ci) Target label - malignant (+ve) or benign (-ve) |X | = 4 |C| = 16 Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Bias Free Learning is Futile A learner making no assumption about target concept cannot classify any unseen instance Inductive Bias Set of assumptions made by a learner to generalize from training examples. Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References Examples of Inductive Bias Rote Learner – No Bias Candidate Elimination – Stronger Bias Find-S – Strongest Bias Varun Chandola Introduction to Machine Learning Concept Learning Learning Conjunctive Concepts Inductive Bias References References W. Maass. On-line learning with an oblivious environment and the power of randomization. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, COLT ’91, pages 167–178, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc. T. M. Mitchell. Machine Learning. McGraw-Hill, Inc., New York, NY, USA, 1 edition, 1997. Varun Chandola Introduction to Machine Learning

Concept Learning in Machine Learning: Find-S Algorithm

Related documents

Products

Support

Concept Learning in Machine Learning: Find-S Algorithm

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib