VISVESVARAYA TECHNOLOGICAL UNIVERSITY
BELAGAVI
ARTIFICIAL INTELLIGENCE AND MACHINE
LEARNING LABORATORY
(Subject Code: 18CSL76)
MASTER MANUAL
VII-SEMESTER (CSE)
AJIET
A J INSTITUTE OF ENGINEERING & TECHNOLOGY
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
(A unit of Laxmi Memorial Education Trust. (R))
NH - 66, Kottara Chowki, Kodical Cross - 575 006
Artificial Intelligence & Machine Learning Laboratory
18CSL76
VISION OF THE INSTITUTE
To produce top-quality engineers who are groomed for attaining excellence in their
profession and competitive enough to help in the growth of nation and global society.
MISSION OF THE INSTITUTE
To offer affordable high-quality graduate program in engineering with value
education and make the students socially responsible.
To support and enhance the institutional environment to attain research excellence in
both faculty and students and to inspire them to push the boundaries of knowledge
base.
To identify the common areas of interest amongst the individuals for the effective
industry-institute partnership in a sustainable way by systematically working
together.
To promote the entrepreneurial attitude and inculcate innovative ideas among the
engineering professionals.
M1
M2
M3
M4
DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING
VISION
To adapt the evolutionary changes in computer science and expose the students to the cutting
edge technologies to produce globally competent professionals.
MISSION
M1
M2
To mould the students by providing quality computer education and by strengthening
the Industry-Academic interface.
To impart the professional skills, innovative research activities and entrepreneurial
capabilities in students.
PROGRAM EDUCATIONAL OBJECTIVES(PEOs)
To develop in students the ability to solve real life problems by applying
fundamental science and elementary strengths of computer science courses.
PEO2 To mould the students to have successful career in IT industry where graduates will
be able to design and implement the needs of society and nation.
PEO3 To transform students to excel in competitive world through higher education and
indulge in research through continuous learning process.
PEO1
Dept. of CS&E, AJIET, Mangalore
1
Artificial Intelligence & Machine Learning Laboratory
18CSL76
PROGRAM OUTCOMES(POs)
PO1
PO2
PO3
PO4
PO5
PO6
PO7
PO8
PO9
PO10
PO11
PO12
Engineering knowledge
Apply the knowledge of mathematics, science, engineering fundamentals and an engineering
specialization to the solution of complex engineering problems.
Problem Analysis
Identify, formulate, review research literature, and analyze complex engineering problems
reaching substantiated conclusions using first principles of mathematics, natural science and
engineering sciences.
Design/development of solutions
Design solutions for complex engineering problems and design system components or processes
that meet the specified needs with appropriate consideration for the public health and safety, and
the cultural, societal and environmental considerations.
Conduct investigations of complex problems
Use research based knowledge and research methods including design of experiments, analysis
and interpretation of data, and synthesis of the information to provide valid conclusions.
Modern tool usage
Create, select and apply appropriate techniques, resources and modern engineering and IT tools
including prediction and modeling to complex engineering activities with an understanding of the
limitations.
The engineer and society
Apply reasoning informed by the contextual knowledge to assess societal, health, safety, legal
and cultural issues and the consequent responsibilities relevant to the professional engineering
practice.
Environment sustainability
Understand the impact of the professional engineering solutions in the societal and environmental
contexts, and demonstrate the knowledge of, and need for sustainable development.
Ethics
Apply ethical principles and commit to professional ethics and responsibilities and norms of the
engineering practice.
Individual and team work
Function effectively as an individual and as a member or leader in diverse teams, and in
multidisciplinary settings.
Communication
Communicate effectively on complex engineering activities with the engineering community and
with society at large, such as, being able to comprehend and write effective reports and design
documentation, make effective presentations, and give and receive clear instructions.
Project management and finance
Demonstrate knowledge and understanding of the engineering and management principles and
apply these to one‟s own work, as a member and leader in a team, to manage projects and in
multidisciplinary environments.
Lifelong learning
Recognize the need for, and have the preparation and ability to engage in independent and
lifelong learning in the broader context of technological change.
PROGRAM SPECIFIC OUTCOMES(PSOs)
PSO1
PSO2
Apply engineering principles, professional ethics and fundamental science in designing
systems and communication models(protocols).
Design and Develop Smart and Intelligent based Applications in Computational
Environment.
Dept. of CS&E, AJIET, Mangalore
2
Artificial Intelligence & Machine Learning Laboratory
18CSL76
University Syllabus
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING LABORATORY
[As per Choice Based Credit System (CBCS) scheme, Effective from the academic year 2018 -2019]
SEMESTER – VII
Laboratory Code
18CSL76
IA Marks
40
Number of Contact Hours/Week
0:0:2
Exam Marks
60
Total Number of Lab Contact Hours
36
Exam Hours
03
CREDITS – 02
Course Learning objectives:
This course will enable students to;
 Implement and Evaluate AI & ML algorithms in Python Programming language.
 Make use of Data sets in implementing the machine learning algorithms
Descriptions (if any)
 Installation procedure of the required software must be demonstrated, carried out in
groups and documented in the journal.
 Data sets can be taken from standard repositories
(https://archive.ics.uci.edu/ml/datasets.html) or constructed by the students.
Course Outcomes:1.
2.
3.
4.
5.
Understand the implementation procedures for the Artificial Intelligence algorithms.
Understand the implementation procedures for the machine learning algorithms.
Design Python programs for various Learning algorithms.
Apply appropriate data sets to the Machine Learning algorithms.
Identify and apply Machine Learning algorithms to solve real world problems.
Dept. of CS&E, AJIET, Mangalore
3
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Laboratory Experiments:
Implement A* Search Algorithm
Implement AO* Search Algorithm.
For a given set of training data examples stored in a .CSV file, implement and
demonstrate the Candidate-Elimination algorithm to output a description of the set of
all hypotheses consistent with the training examples.
Write a program to demonstrate the working of the decision tree based ID3 algorithm.
Use an appropriate data set for building the decision tree and apply this knowledge to
classify a new sample.
Build an Artificial Neural Network by implementing the Backpropagation algorithm
and test the same using appropriate data sets.
Write a program to implement the naïve Bayesian classifier for a sample training data
set stored as a .CSV file. Compute the accuracy of the classifier, considering few test
data sets.
Apply EM algorithm to cluster a set of data stored in a .CSV file. Use the same data
set for clustering using k-Means algorithm. Compare the results of these two
algorithms and comment on the quality of clustering. You can add Java/Python ML
library classes/API in the program.
Write a program to implement k-Nearest Neighbour algorithm to classify the iris data
set. Print both correct and wrong predictions. Java/Python ML Library classes can be
used for this problem.
Implement the non-parametric Locally Weighted Regression algorithm in order to fit
data points. Select appropriate data set for your experiment and draw graphs.
1
2
3
4
5
6
7
8
9
Course outcomes:
On the completion of this laboratory course, the students will be able to:
1. Understand the implementation procedures for the Artificial Intelligence algorithms.
2. Understand the implementation procedures for the machine learning algorithms.
3. Design Python programs for various Learning algorithms.
4. Apply appropriate data sets to the Machine Learning algorithms.
5. Identify and apply Machine Learning algorithms to solve real world problems.
Graduate Attributes (as per NBA)
1. Engineering Knowledge
2. Problem Analysis
3. Design/Development of Solutions
4. Modern Tool Usage
Conduction of Practical Examination:
 Experiment distribution
o For laboratories having only one part: Students are allowed to pick one
experiment from the lot with equal opportunity.

Change of experiment is allowed only once and marks allotted for procedure to be
made zero of the changed part only.

Marks Distribution:
o For laboratories having one part
- Procedure + Execution + Viva –Voce = 15 + 70 + 15 = 100 Marks
Dept. of CS&E, AJIET, Mangalore
4
Artificial Intelligence & Machine Learning Laboratory
18CSL76
PROGRAM 1
Q.1. Implement A* Search Algorithm
Theory:
from collections import deque
Deque (Doubly Ended Queue) in Python is implemented using the module “collections“.
Deque is preferred over list in the cases where we need quicker append and pop operations
from both the ends of container.
various Operations on deque :
 append() :- This function is used to insert the value in its argument to the right end of
deque.
 appendleft() :- This function is used to insert the value in its argument to the left
end of deque.
 pop() :- This function is used to delete an argument from the right end of deque.
 popleft() :- This function is used to delete an argument from the left end of deque.
def __init__(self, adjac_lis):
The __init__ method is similar to constructors in C++ and Java. Constructors are used to
initialize the object‟s state.
The keyword self represents the instance of a class and binds the attributes with the given
arguments.





A* is an optimal shortest path algorithm.
It is the best form of BFS(Breath First Search).
It‟s like Dijkstras algorithm, but with a heuristic function.
It is used in many Games & Web-based maps.
A* search algorithm uses the following;
g(n) -> Cost of the path from start node to „n‟ node.
h(n) -> Heuristic function that estimates the cost of the cheapest path from „n‟ to the
goal node.
Algorithm:
Step 1: Firstly, Place the starting node into OPEN and find its f (n) value.
Step 2: Then remove the node from OPEN, having the smallest f (n) value. If it is a
goal node, then stop and return to success.
Step 3: Else remove the node from OPEN, and find all its successors.
Step 4: Find the f (n) value of all the successors, place them into OPEN, and place
the removed node into CLOSE.
Step 5: Go to Step-2.
Step 6: Exit.
Dept. of CS&E, AJIET, Mangalore
5
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Program 1
from collections import deque
class Graph:
def __init__(self, adjac_lis):
self.adjac_lis = adjac_lis
def get_neighbors(self, v):
return self.adjac_lis[v]
# This is heuristic function which is having equal values for all nodes
def h(self, n):
H={
'S': 5,
'A': 3,
'B': 4,
'C': 2,
'D': 6,
'G': 0
}
return H[n]
def a_star_algorithm(self, start, stop):
# In this open_lst is a list of nodes which have been visited, but who's
# neighbours haven't all been always inspected, It starts off with the
# start node
# And closed_lst is a list of nodes which have been visited
# and who's neighbors have been always inspected
open_lst = set([start])
closed_lst = set([])
# poo has present distances from start to all other nodes
# the default value is +infinity
poo = {}
poo[start] = 0
# par contains an adjac mapping of all nodes
par = {}
par[start] = start
while len(open_lst) > 0:
n = None
# it will find a node with the lowest value of f() for v in open_lst:
if n == None or poo[v] + self.h(v) < poo[n] + self.h(n):
n=v
if n == None:
print('Path does not exist!')
return None
Dept. of CS&E, AJIET, Mangalore
6
Artificial Intelligence & Machine Learning Laboratory
18CSL76
# if the current node is the stop then we start again from start
if n == stop:
reconst_path = []
while par[n] != n:
reconst_path.append(n)
n = par[n]
reconst_path.append(start)
reconst_path.reverse()
print('Path found: {}'.format(reconst_path))
return reconst_path
# for all the neighbors of the current node do
for (m, weight) in self.get_neighbors(n):
# if the current node is not presentin both open_lst and closed_lst
# add it to open_lst and note n as it's par
if m not in open_lst and m not in closed_lst:
open_lst.add(m)
par[m] = n
poo[m] = poo[n] + weight
# otherwise, check if it's quicker to first visit n, then m
# and if it is, update par data and poo data
# and if the node was in the closed_lst, move it to open_lst
else:
if poo[m] > poo[n] + weight:
poo[m] = poo[n] + weight
par[m] = n
if m in closed_lst:
closed_lst.remove(m)
open_lst.add(m)
# remove n from the open_lst, and add it to closed_lst because all of his neighbors were inspected
open_lst.remove(n)
closed_lst.add(n)
print('Path does not exist!')
return None
adjac_lis = {
'S': [('A',1),('G',10)],
'A': [('B', 2), ('C', 1)],
'B': [('D', 3)],
'C': [('D', 3),('G',4)],
'D': [('G', 2)]
}
graph1 = Graph(adjac_lis)
graph1.a_star_algorithm('S', 'G')
Dept. of CS&E, AJIET, Mangalore
7
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Output Examples
Example Graph
Output will be as follows;
Viva Questions:
1. Give any application of A* Search algorithm.
Answer: It is used in many Games & Web-based maps.
2. Why A* search algorithm is used?
Answer: It is used to find the optimal shortest path from starting node to destination
node.
3. What is heuristic function (or) the heuristic value that is used in A* Search ?
Answer: A heuristic function or a heuristic is a shortcut to solving a problem when
there are no exact solutions for it (or) the time to obtain the solution is too long.
Dept. of CS&E, AJIET, Mangalore
8
Artificial Intelligence & Machine Learning Laboratory
18CSL76
PROGRAM 2
Q.2. Implement AO* Search Algorithm.
Theory
 AO* Algorithm is a generalized algorithm, which will always find minimum cost
solution.
 It is used for solving cyclic AND-OR graphs The AO* will use a single structure
GRAPH representing the part of the search graph that has been explicitly generated so
far.
 Each node in the graph will point both down to its immediate successors and up to
immediate predecessors.
 The top down traversing of the best-known path which guarantees that only nodes that
are on the best path will ever be considered for expansion.
Algorithm:
Step 1: Place the starting node into OPEN.
Step 2: Compute the most promising solution tree say T0.
Step 3: Select a node n that is both on OPEN and a member of T0. Remove it from OPEN and
place it in CLOSE
Step 4: If n is the terminal goal node then leveled n as solved and leveled all the ancestors of n
as solved. If the starting node is marked as solved then success and exit.
Step 5: If n is not a solvable node, then mark n as unsolvable. If starting node is marked as
unsolvable, then return failure and exit.
Step 6: Expand n. Find all its successors and find their h (n) value, push them into OPEN.
Step 7: Return to Step 2.
Step 8: Exit.
Program 2:
# Recursive implementation of AO* algorithm
class Graph:
#instantiate graph object with graph topology, heuristic values, start node
def __init__(self, graph, heuristicNodeList, startNode):
Dept. of CS&E, AJIET, Mangalore
9
Artificial Intelligence & Machine Learning Laboratory
18CSL76
self.graph = graph
self.H=heuristicNodeList
self.start=startNode
self.parent={}
self.status={}
self.solutionGraph={}
# starts a recursive AO* algorithm
def applyAOStar(self):
self.aoStar(self.start, False)
# gets the Neighbors of a given node
def getNeighbors(self, v):
return self.graph.get(v,'')
# return the status of a given node
def getStatus(self,v):
return self.status.get(v,0)
# set the status of a given node
def setStatus(self,v, val):
self.status[v]=val
def getHeuristicNodeValue(self, n):
return self.H.get(n,0) # always return the heuristic value of a given node
def setHeuristicNodeValue(self, n, value):
self.H[n]=value
# set the revised heuristic value of a given node
def printSolution(self):
print("FOR GRAPH SOLUTION, TRAVERSE THE GRAPH FROM THE START
NODE:",self.start)
print("------------------------------------------------------------")
print(self.solutionGraph)
print("------------------------------------------------------------")
# Computes the Minimum Cost of child nodes of a given node v
def computeMinimumCostChildNodes(self, v):
minimumCost=0
costToChildNodeListDict={}
costToChildNodeListDict[minimumCost]=[]
flag=True
for nodeInfoTupleList in self.getNeighbors(v): # iterate over all the set of child nodes
cost=0
Dept. of CS&E, AJIET, Mangalore
10
Artificial Intelligence & Machine Learning Laboratory
18CSL76
nodeList=[]
for c, weight in nodeInfoTupleList:
cost=cost+self.getHeuristicNodeValue(c)+weight
nodeList.append(c)
# initialize Minimum Cost with the cost of first set of child nodes
if flag==True:
minimumCost=cost
costToChildNodeListDict[minimumCost]=nodeList # set the Minimum Cost child nodes
flag=False
else: # checking the Minimum Cost nodes with the current Minimum Cost
if minimumCost>cost:
minimumCost=cost
costToChildNodeListDict[minimumCost]=nodeList # set the Minimum Cost child node
# return Minimum Cost and Minimum Cost child nodes
return minimumCost, costToChildNodeListDict[minimumCost]
# AO* algorithm for a start node and backTracking status flag
def aoStar(self, v, backTracking):
if self.getStatus(v) >= 0: # if status node v >= 0, compute Minimum Cost nodes of v
minimumCost, childNodeList = self.computeMinimumCostChildNodes(v)
self.setHeuristicNodeValue(v, minimumCost)
self.setStatus(v,len(childNodeList))
solved=True # check the Minimum Cost nodes of v are solved
for childNode in childNodeList:
self.parent[childNode]=v
if self.getStatus(childNode)!=-1:
solved=solved & False
#if the Minimum Cost nodes of v are solved, set the current node status as solved(-1)
if solved==True:
self.setStatus(v,-1)
self.solutionGraph[v]=childNodeList
# update solution graph with solved nodes which may be a part of solution
# check the current node is the start node for backtracking the current node value
if v!=self.start:
self.aoStar(self.parent[v], True)
# backtracking the current node value with backtracking status set to true
if backTracking==False:
# check the current call is not for backtracking
for childNode in childNodeList: # for each Minimum Cost child node
self.setStatus(childNode,0) # set the status of child node to 0
self.aoStar(childNode, False)
# Minimum Cost child node is further explored with backtracking status as false
Dept. of CS&E, AJIET, Mangalore
11
Artificial Intelligence & Machine Learning Laboratory
18CSL76
h = {'A': 1, 'B': 6, 'C': 12, 'D': 10, 'E': 4, 'F': 4, 'G': 5, 'H': 7} # Heuristic values of Nodes
graph = {
# Graph of Nodes and Edges
'A': [[('B', 1), ('C', 1)], [('D', 1)]],
'B': [[('G', 1)], [('H', 1)]],
'D': [[('E', 1), ('F', 1)]]
}
G = Graph(graph, h, 'A') # Instantiate Graph object with graph, heuristic values and start Node
G.applyAOStar()
# Run the AO* algorithm
G.printSolution()
# Print the solution graph as output of the AO* algorithm search
Output Example
Example Graph
Output obtained
Viva Questions:
1. What is AO* Search algorithm ?
AO* Algorithm is a generalized algorithm, which will always find minimum cost
solution.
2. Will AO* search consider all the nodes in the graph ?
No, AO* search considers only selected nodes which satisfies the requirement, and
will ignore the other nodes which doesn‟t satisfy the requirement in search process.
Dept. of CS&E, AJIET, Mangalore
12
Artificial Intelligence & Machine Learning Laboratory
18CSL76
PROGRAM 3:
Q.3. For a given set of training data examples stored in a .CSV file, implement and
demonstrate the Candidate-Elimination algorithm to output a description of the
set of all hypotheses consistent with the training examples.
Theory
The candidate elimination algorithm incrementally builds the version space given a
hypothesis space H and a set E of examples. The examples are added one by one; each
example possibly shrinks the version space by removing the hypotheses that are
inconsistent with the example. The candidate elimination algorithm does this by updating
the general and specific boundary for each new example.




You can consider this as an extended form of Find-S algorithm.
Consider both positive and negative examples.
Actually, positive examples are used here as Find-S algorithm (Basically they are
generalizing from the specification).
While the negative example is specified from generalize form.
Terms Used:






Concept learning: Concept learning is basically learning task of the machine (Learn by
Train data)
General Hypothesis: Not Specifying features to learn the machine.
G = {‘?’, ‘?’,’?’,’?’…}: Number of attributes
Specific Hypothesis: Specifying features to learn machine (Specific feature)
S= {‘pi’,’pi’,’pi’…}: Number of pi depends on number of attributes.
Version Space: It is intermediate of general hypothesis and Specific hypothesis. It not
only just written one hypothesis but a set of all possible hypothesis based on training
data-set.
Algorithm:
Step1: Load Data set
Step2: Initialize General Hypothesis and Specific Hypothesis.
Step3: For each training example
Step4: If example is positive example
If attribute_value == hypothesis_value:
Do nothing
else:
replace attribute value with '?'(Basically generalizing it)
Step5: If example is Negative example
Make generalize hypothesis more specific.
Dept. of CS&E, AJIET, Mangalore
13
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Program 3:
import numpy as np
import pandas as pd
# Loading Data from a CSV File
data = pd.DataFrame(data=pd.read_csv('EnjoySport.csv'))
# Separating concept features from Target
concepts = np.array(data.iloc[:,0:-1])
print(concepts)
# Isolating target into a separate DataFrame
#copying last column to target array
target = np.array(data.iloc[:,-1])
def learn(concepts, target):
# Initialise S0 with the first instance from concepts
# .copy() makes sure a new list is created instead of just pointing to the same memory
#location
print("initialization of specific_h and general_h")
l=len(concepts[0])
specific_h=['0']*l
print(specific_h)
general_h=['?']*l
print(general_h,"\n")
specific_h = concepts[0].copy()
#print("initialization of specific_h and general_h")
#print(specific_h)
general_h = [["?" for i in range(len(specific_h))] for i in range(len(specific_h))]
#print(general_h)
# The learning iterations
for i, h in enumerate(concepts):
# Checking if the hypothesis has a positive target
if target[i] == "Yes":
for x in range(len(specific_h)):
# Change values in S & G only if values change
if h[x] != specific_h[x]:
specific_h[x] = '?'
general_h[x][x] = '?'
# Checking if the hypothesis has a positive target
if target[i] == "No":
for x in range(len(specific_h)):
# For negative hyposthesis change values only in G
if h[x] != specific_h[x]:
Dept. of CS&E, AJIET, Mangalore
14
Artificial Intelligence & Machine Learning Laboratory
18CSL76
general_h[x][x] = specific_h[x]
else:
general_h[x][x] = '?'
print(" steps ", i+1 , " of Candidate Elimination Algorithm")
print(specific_h)
print(general_h)
# find indices where we have empty rows, meaning those that are unchanged
indices = [i for i, val in enumerate(general_h) if val == ['?', '?', '?', '?', '?', '?']]
for i in indices:
# remove those rows from general_h
general_h.remove(['?', '?', '?', '?', '?', '?'])
# Return final values
return specific_h, general_h
s_final, g_final = learn(concepts, target)
print("\nFinal Specific_h:", s_final, sep="\n")
print("Final General_h:", g_final, sep="\n")
Output Example
Consider the dataset given below:
Algorithmic steps:
Initially : G =[[?, ?, ?, ?, ?, ?], [?, ?, ?, ?, ?, ?], [?, ?, ?, ?, ?, ?], [?, ?, ?, ?, ?, ?], [?, ?, ?, ?, ?, ?],
[?, ?, ?, ?, ?, ?]]
S = [Null, Null, Null, Null, Null, Null]
For instance 1 : <'sunny','warm','normal','strong','warm ','same'> and positive output.
G1 = G
S1 = ['sunny','warm','normal','strong','warm ','same']
For instance 2 :<'sunny','warm','high','strong','warm ','same'> and positive output.
G2 = G
S2 = ['sunny','warm',?,'strong','warm ','same']
Dept. of CS&E, AJIET, Mangalore
15
Artificial Intelligence & Machine Learning Laboratory
18CSL76
For instance 3 :<'rainy','cold','high','strong','warm ','change'> and negative output.
G3 = [['sunny', ?, ?, ?, ?, ?], [?, 'warm', ?, ?, ?, ?], [?, ?, ?, ?, ?, ?], [?, ?, ?, ?, ?, ?], [?, ?, ?, ?, ?, ?],
[?, ?, ?, ?, ?, 'same']]
S3 = S2
For instance 4 :<'sunny','warm','high','strong','cool','change'> and positive output.
G4 = G3
S4 = ['sunny','warm',?,'strong', ?, ?]
At last, by synchronizing the G4 and S4 algorithm produce the output.
Output obtained
initialization of specific_h and general_h
['0', '0', '0', '0', '0', '0']
['?', '?', '?', '?', '?', '?']
steps 1 of Candidate Elimination Algorithm
['Sunny' 'Warm' 'Normal' 'Strong' 'Warm' 'Same']
[['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'],
['?', '?', '?', '?', '?', '?']]
steps 2 of Candidate Elimination Algorithm
['Sunny' 'Warm' '?' 'Strong' 'Warm' 'Same']
[['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'],
['?', '?', '?', '?', '?', '?']]
steps 3 of Candidate Elimination Algorithm
['Sunny' 'Warm' '?' 'Strong' 'Warm' 'Same']
[['Sunny', '?', '?', '?', '?', '?'], ['?', 'Warm', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '
?', '?', '?'], ['?', '?', '?', '?', '?', 'Same']]
steps 4 of Candidate Elimination Algorithm
['Sunny' 'Warm' '?' 'Strong' '?' '?']
[['Sunny', '?', '?', '?', '?', '?'], ['?', 'Warm', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '
?', '?', '?'], ['?', '?', '?', '?', '?', '?']]
Final Specific_h:
['Sunny' 'Warm' '?' 'Strong' '?' '?']
Final General_h:
[['Sunny', '?', '?', '?', '?', '?'], ['?', 'Warm', '?', '?', '?', '?']]
Viva Questions:
1. What do you mean by hypotheses ?
Answer: A supposition or proposed explanation made on the basis of limited evidence
as a starting point for further investigation.
Dept. of CS&E, AJIET, Mangalore
16
Artificial Intelligence & Machine Learning Laboratory
18CSL76
2. What is general hypothesis and specific hypothesis ?
Answer: The general hypothesis states the general relationship between the major
variables. Put simply, a hypothesis is a specific, testable prediction. More specifically,
it describes in concrete terms what you expect will happen in a certain circumstance.
A hypothesis is used to determine the relationship between two variables, which are
the two things that are being tested.
3. What is a concept ?
Answer: Concept learning describes the process by which experience allows us to
partition objects in the world into classes for the purpose of generalization,
discrimination, and inference.
4. What is classification?
In machine learning, classification refers to a predictive modelling problem where a
class label is predicted for a given example of input data.
Dept. of CS&E, AJIET, Mangalore
17
Artificial Intelligence & Machine Learning Laboratory
18CSL76
PROGRAM 4
Q.4. Write a program to demonstrate the working of the decision tree based ID3
algorithm. Use an appropriate data set for building the decision tree and apply
this knowledge to classify a new sample.
Theory
ID3 stands for Iterative Dichotomiser 3. It is a classification algorithm that follows a greedy
approach by selecting a best attribute that yields maximum Information Gain (IG) or
minimum Entropy(H).
Entropy is a measure of the amount of uncertainty in the dataset S. Mathematical
Representation of Entropy is shown here H(S)=∑c∈C−p(c)log2p(c)
Where,
 S - The current dataset for which entropy is being calculated(changes every iteration
of the ID3 algorithm).
 C - Set of classes in S {example - C ={yes, no}}
 p(c) - The proportion of the number of elements in class c to the number of elements
in set S.
In ID3, entropy is calculated for each remaining attribute. The attribute with the smallest
entropy is used to split the set S on that particular iteration.
Entropy = 0 implies it is of pure class, that means all are of same category.
Information Gain IG(A) tells us how much uncertainty in S was reduced after splitting set S
on attribute A. Mathematical representation of Information gain is shown here IG(A,S)=H(S)−∑t∈ Tp(t)H(t)
Where,
 H(S) - Entropy of set S.
 T - The subsets created from splitting set S by attribute A such that
S=⋃tϵTt
 p(t) - The proportion of the number of elements in t to the number of elements in set S.
 H(t) - Entropy of subset t.
In ID3, information gain can be calculated (instead of entropy) for each remaining attribute.
The attribute with the largest information gain is used to split the set S on that particular
iteration.
Dept. of CS&E, AJIET, Mangalore
18
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Algorithm:
Step 1: Calculate entropy for dataset.
Step 2: For each attribute/feature.
2.1. Calculate entropy for all its categorical values.
2.2. Calculate information gain for the feature.
Step 3: Find the feature with maximum information gain.
Step 4: Repeat it until we get the desired tree.
Program 4:
import pandas as pd
import numpy as np
dataset=pd.read_csv('Tennis.csv',names=['Outlook','Temperature','Humidity','Wind','PlayTennis'])
def entropy(target_col):
elements,counts = np.unique(target_col,return_counts = True)
entropy = np.sum([(-counts[i]/np.sum(counts))*np.log2(counts[i]/np.sum(counts)) for i in
range(len(elements))])
return entropy
def InfoGain(data,split_attribute_name,target_name="PlayTennis"):
total_entropy = entropy(data[target_name])
vals,counts= np.unique(data[split_attribute_name],return_counts=True)
Weighted_Entropy =
np.sum([(counts[i]/np.sum(counts))*entropy(data.where(data[split_attribute_name]==vals[i]).drop
na()[target_name]) for i in range(len(vals))])
Information_Gain = total_entropy - Weighted_Entropy
return Information_Gain
def ID3(data,originaldata,features,target_attribute_name="PlayTennis",parent_node_class = None):
if len(np.unique(data[target_attribute_name])) <= 1:
return np.unique(data[target_attribute_name])[0]
elif len(data)==0:
return
np.unique(originaldata[target_attribute_name])[np.argmax(np.unique(originaldata[target_attribute
_name],return_counts=True)[1])]
elif len(features) ==0:
return parent_node_class
else:
parent_node_class =
np.unique(data[target_attribute_name])[np.argmax(np.unique(data[target_attribute_name],return_
counts=True)[1])]
item_values = [InfoGain(data,feature,target_attribute_name) for feature in features]
#Return the information gain values for the features in the dataset
best_feature_index = np.argmax(item_values)
best_feature = features[best_feature_index]
tree = {best_feature:{}}
features = [i for i in features if i != best_feature]
for value in np.unique(data[best_feature]):
Dept. of CS&E, AJIET, Mangalore
19
Artificial Intelligence & Machine Learning Laboratory
18CSL76
value = value
sub_data = data.where(data[best_feature] == value).dropna()
subtree = ID3(sub_data,dataset,features,target_attribute_name,parent_node_class)
tree[best_feature][value] = subtree
return(tree)
tree = ID3(dataset,dataset,dataset.columns[:-1])
print(' Display Tree\n',tree)
Output Example
Final Decision Tree will be as follows;
Data Set
Here, dataset is of binary classes (yes and no), where 9 out of 14 are "yes" and 5 out of 14 are "no".
Dept. of CS&E, AJIET, Mangalore
20
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Complete entropy of dataset is:
H(S) = - p(yes) * log2(p(yes)) - p(no) * log2(p(no))
= - (9/14) * log2(9/14) - (5/14) * log2(5/14)
= - (-0.41) - (-0.53)
= 0.94
For each attribute of the dataset, let's follow the step-2 of pseudocode : -
First Attribute - Outlook
Categorical values - sunny, overcast and rain
H(Outlook=sunny)= -(2/5)*log(2/5)-(3/5)*log(3/5) =0.971
H(Outlook=rain) = -(3/5)*log(3/5)-(2/5)*log(2/5) =0.971
H(Outlook=overcast) = -(4/4)*log(4/4)-0 = 0
Average Entropy Information for Outlook I(Outlook) = p(sunny)
* H(Outlook=sunny)
+ p(rain) * H(Outlook=rain)
+ p(overcast) *
H(Outlook=overcast)
= (5/14)*0.971 + (5/14)*0.971 + (4/14)*0
= 0.693
Information Gain = H(S) - I(Outlook)
= 0.94 - 0.693
= 0.247
Second Attribute - Temperature
Categorical values - hot, mild, cool
H(Temperature=hot)= -(2/4)*log(2/4)-(2/4)*log(2/4) = 1
H(Temperature=cool) = -(3/4)*log(3/4)-(1/4)*log(1/4) = 0.811
H(Temperature=mild) = -(4/6)*log(4/6)-(2/6)*log(2/6) = 0.9179
Average Entropy Information for Temperature I(Temperature)
=
p(hot)*H(Temperature=hot)
+
p(mild)*H(Temperature=mild)
+
p(cool)*H(Temperature=cool)
= (4/14)*1 + (6/14)*0.9179 + (4/14)*0.811
= 0.9108
Information Gain = H(S) - I(Temperature)
= 0.94 - 0.9108
Dept. of CS&E, AJIET, Mangalore
21
Artificial Intelligence & Machine Learning Laboratory
18CSL76
= 0.0292
Third Attribute - Humidity
Categorical values - high, normal
H(Humidity=high)= -(3/7)*log(3/7)-(4/7)*log(4/7) = 0.983
H(Humidity=normal) = -(6/7)*log(6/7)-(1/7)*log(1/7) = 0.591
Average Entropy Information for Humidity I(Humidity) = p(high)*H(Humidity=high) + p(normal)*H(Humidity=normal)
= (7/14)*0.983 + (7/14)*0.591
= 0.787
Information Gain = H(S) - I(Humidity)
= 0.94 - 0.787
= 0.153
Fourth Attribute - Wind
Categorical values - weak, strong
H(Wind=weak) = -(6/8)*log(6/8)-(2/8)*log(2/8) = 0.811
H(Wind=strong) = -(3/6)*log(3/6)-(3/6)*log(3/6) = 1
Average Entropy Information for Wind I(Wind) = p(weak)*H(Wind=weak) + p(strong)*H(Wind=strong)
= (8/14)*0.811 + (6/14)*1
= 0.892
Information Gain = H(S) - I(Wind)
= 0.94 - 0.892
= 0.048
Here, the attribute with maximum information gain is Outlook. So, the decision tree built so
far -
Dept. of CS&E, AJIET, Mangalore
22
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Here,
when
Outlook
==
overcast,
it
is
of
pure
class(Yes).
Now, we have to repeat same procedure for the data with rows consist of Outlook value as
Sunny and then for Outlook value as Rain.
Now, finding the best attribute for splitting the data with Outlook=Sunny values{ Dataset
rows = [1, 2, 8, 9, 11]}.
Complete entropy of Sunny is H(S) = - p(yes) * log2(p(yes)) - p(no) * log2(p(no))
= - (2/5) * log2(2/5) - (3/5) * log2(3/5)
= 0.971
First Attribute - Temperature
Categorical values - hot, mild, cool
H(Sunny, Temperature=hot)= -0-(2/2)*log(2/2) = 0
H(Sunny, Temperature=cool) = -(1)*log(1)- 0 = 0
H(Sunny, Temperature=mild) = -(1/2)*log(1/2)-(1/2)*log(1/2) = 1
Average Entropy Information for Temperature I(Sunny, Temperature) = p(Sunny, hot)*H(Sunny, Temperature=hot) + p(Sunny, mild)*H(Sunny,
Temperature=mild) + p(Sunny, cool)*H(Sunny, Temperature=cool)
= (2/5)*0 + (1/5)*0 + (2/5)*1
= 0.4
Information Gain = H(Sunny) - I(Sunny, Temperature)
= 0.971 - 0.4
= 0.571
Second Attribute - Humidity
Categorical values - high, normal
H(Sunny, Humidity=high)= - 0 - (3/3)*log(3/3) = 0
Dept. of CS&E, AJIET, Mangalore
23
Artificial Intelligence & Machine Learning Laboratory
18CSL76
H(Sunny, Humidity=normal) = -(2/2)*log(2/2)-0 = 0
Average Entropy Information for Humidity I(Sunny, Humidity) = p(Sunny, high)*H(Sunny, Humidity=high) + p(Sunny, normal)*H(Sunny,
Humidity=normal)
= (3/5)*0 + (2/5)*0
=0
Information Gain = H(Sunny) - I(Sunny, Humidity)
= 0.971 - 0
= 0.971
Third Attribute - Wind
Categorical values - weak, strong
H(Sunny, Wind=weak) = -(1/3)*log(1/3)-(2/3)*log(2/3) = 0.918
H(Sunny, Wind=strong) = -(1/2)*log(1/2)-(1/2)*log(1/2) = 1
Average Entropy Information for Wind I(Sunny, Wind) = p(Sunny, weak)*H(Sunny, Wind=weak) + p(Sunny, strong)*H(Sunny, Wind=strong)
= (3/5)*0.918 + (2/5)*1
= 0.9508
Information Gain = H(Sunny) - I(Sunny, Wind)
= 0.971 - 0.9508
= 0.0202
Here, the attribute with maximum information gain is Humidity. So, the decision tree built so far –
Dept. of CS&E, AJIET, Mangalore
24
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Here, when Outlook = Sunny and Humidity = High, it is a pure class of category "no". And
When Outlook = Sunny and Humidity = Normal, it is again a pure class of category "yes".
Therefore, we don't need to do further calculations.
Now, finding the best attribute for splitting the data with Outlook=Rain values{ Dataset rows
= [4, 5, 6, 10, 14]}.
Complete entropy of Rain is H(S) = - p(yes) * log2(p(yes)) - p(no) * log2(p(no))
= - (3/5) * log(3/5) - (2/5) * log(2/5)
= 0.971
First Attribute - Temperature
Categorical values - mild, cool
H(Rain, Temperature=cool)= -(1/2)*log(1/2)- (1/2)*log(1/2) = 1
H(Rain, Temperature=mild) = -(2/3)*log(2/3)-(1/3)*log(1/3) = 0.918
Average Entropy Information for Temperature I(Rain,
Temperature)
=
p(Rain,
mild)*H(Rain,
Temperature=mild)
+
p(Rain,
cool)*H(Rain,
Temperature=cool)
= (2/5)*1 + (3/5)*0.918
= 0.9508
Information Gain = H(Rain) - I(Rain, Temperature)
= 0.971 - 0.9508
= 0.0202
Dept. of CS&E, AJIET, Mangalore
25
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Second Attribute - Wind
Categorical values - weak, strong
H(Wind=weak) = -(3/3)*log(3/3)-0 = 0
H(Wind=strong) = 0-(2/2)*log(2/2) = 0
Average Entropy Information for Wind I(Wind) = p(Rain, weak)*H(Rain, Wind=weak) + p(Rain, strong)*H(Rain, Wind=strong)
= (3/5)*0 + (2/5)*0
=0
Information Gain = H(Rain) - I(Rain, Wind)
= 0.971 - 0
= 0.971
Here, the attribute with maximum information gain is Wind. So, the decision tree built so far -
Here, when Outlook = Rain and Wind = Strong, it is a pure class of category "no". And When
Outlook = Rain and Wind = Weak, it is again a pure class of category "yes".
And this is our final desired tree for the given dataset.
Output Obtained is as follows
Display Tree
{'Outlook': {'Overcast': 'Yes', 'Rain': {'Wind': {'Strong': 'No', 'Weak': 'Yes'}}, 'Sunny': {'Humidity':
{'High': 'No', 'Normal': 'Yes'}}}}
Viva Questions:
1.
Define target function.
Answer: A target function, in machine learning, is a method for solving a problem
that an AI algorithm parses its training data to find. Once an algorithm finds its
target function, that function can be used to predict results.
Dept. of CS&E, AJIET, Mangalore
26
Artificial Intelligence & Machine Learning Laboratory
18CSL76
2.
Define decision tree.
Answer: Decision Trees are a type of Supervised Machine Learning (that is you
explain what the input is and what the corresponding output is in the training data)
where the data is continuously split according to a certain parameter. The tree can be
explained by two entities, namely decision nodes and leaves.
3.
Define entropy.
Answer: Entropy known as the controller for decision tree to decide where to split
the data.
Dept. of CS&E, AJIET, Mangalore
27
Artificial Intelligence & Machine Learning Laboratory
18CSL76
PROGRAM 5:
Q.5. Build an Artificial Neural Network by implementing the Back-propagation
algorithm and test the same using appropriate data sets.
Theory
 Back-Propagation algorithm is a supervised learning algorithm for training Neural
Network.
 The back-propagation algorithm performs learning on a multilayer feed-forward neural
network.
 It iteratively learns a set of weights for prediction of the class label of tuples.
 A multilayer feed-forward neural network consists of an input layer, one or more hidden
layers, and an output layer.
 An example of a multilayer feed-forward network is shown in the figure below;
Each layer is made up of units. The inputs to the network correspond to the attributes
measured for each training tuple. The inputs are fed simultaneously into the units making
up the input layer.
These inputs pass through the input layer and are then weighted and fed simultaneously
to a second layer of “neuronlike” units, known as a hidden layer. The outputs of the
hidden layer units can be input to another hidden layer, and so on.
The number of hidden layers is arbitrary, although in practice, usually only one is used.
The weighted outputs of the last hidden layer are input to units making up the output
layer, which emits the network's prediction for given tuples.
Dept. of CS&E, AJIET, Mangalore
28
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Algorithm:
Program 5:
import numpy as np
X = np.array(([2, 9], [1, 5], [3, 6]), dtype=float)
y = np.array(([92], [86], [89]), dtype=float)
X = X/np.amax(X,axis=0) # maximum of X array longitudinally
y = y/100
#Sigmoid Function
def sigmoid (x):
return 1/(1 + np.exp(-x))
#Derivative of Sigmoid Function
def derivatives_sigmoid(x):
return x * (1 - x)
#Variable initialization
epoch=5000 #Setting training iterations
lr=0.1
#Setting learning rate
inputlayer_neurons = 2
#number of features in data set
hiddenlayer_neurons = 3
#number of hidden layers neurons
output_neurons = 1
#number of neurons at output layer
#weight and bias initialization
wh=np.random.uniform(size=(inputlayer_neurons,hiddenlayer_neurons))
bh=np.random.uniform(size=(1,hiddenlayer_neurons))
wout=np.random.uniform(size=(hiddenlayer_neurons,output_neurons))
bout=np.random.uniform(size=(1,output_neurons))
Dept. of CS&E, AJIET, Mangalore
29
Artificial Intelligence & Machine Learning Laboratory
18CSL76
#draws a random range of numbers uniformly of dim x*y
for i in range(epoch):
#Forward Propogation
hinp1=np.dot(X,wh)
hinp=hinp1 + bh
hlayer_act = sigmoid(hinp)
outinp1=np.dot(hlayer_act,wout)
outinp= outinp1+ bout
output = sigmoid(outinp)
#Backpropagation
EO = y-output
outgrad = derivatives_sigmoid(output)
d_output = EO* outgrad
EH = d_output.dot(wout.T)
#how much hidden layer wts contributed to error
hiddengrad = derivatives_sigmoid(hlayer_act)
d_hiddenlayer = EH * hiddengrad
# dotproduct of nextlayererror and currentlayerop
wout += hlayer_act.T.dot(d_output) *lr
wh += X.T.dot(d_hiddenlayer) *lr
print("Input: \n" + str(X))
print("Actual Output: \n" + str(y))
print("Predicted Output: \n" ,output)
Output Example
Input:
[[0.66666667 1.
]
[0.33333333 0.55555556]
[1.
0.66666667]]
Actual Output:
[[0.92]
[0.86]
[0.89]]
Predicted Output:
[[0.73343398]
[0.71979816]
[0.73388742]]
Dept. of CS&E, AJIET, Mangalore
30
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Viva Questions:
1. What is ANN?
Answer: An artificial neuron network (neural network) is a computational model that
mimics the way nerve cells work in the human brain. Artificial neural networks
(ANNs) use learning algorithms that can independently make adjustments - or learn,
in a sense - as they receive new input.
2. Define single layer perceptron.
Answer: A single layer perceptron (SLP) is a feed-forward network based on a
threshold transfer function. SLP is the simplest type of artificial neural networks and
can only classify linearly separable cases with a binary target (1 , 0).
3. What is back propagation?
Answer: Essentially, backpropagation is an algorithm used to calculate derivatives
quickly. Artificial neural networks use backpropagation as a learning algorithm to
compute a gradient descent with respect to weights.
Dept. of CS&E, AJIET, Mangalore
31
Artificial Intelligence & Machine Learning Laboratory
18CSL76
PROGRAM 6
Q.6. Write a program to implement the naïve Bayesian classifier for a sample training
data set stored as a .CSV file. Compute the accuracy of the classifier, considering
few test data sets.
Theory
It is a classification technique based on Bayes‟ Theorem with an assumption of independence
among predictors. In simple terms, a Naive Bayes classifier assumes that the presence of a
particular feature in a class is unrelated to the presence of any other feature.
For example, a fruit may be considered to be an apple if it is red, round, and about 3 inches in
diameter. Even if these features depend on each other or upon the existence of the other
features, all of these properties independently contribute to the probability that this fruit is an
apple and that is why it is known as „Naive‟.
Naive Bayes model is easy to build and particularly useful for very large data sets. Along
with simplicity, Naive Bayes is known to outperform even highly sophisticated classification
methods.
Bayes theorem provides a way of calculating posterior probability P(c|x) from P(c), P(x) and
P(x|c). Look at the equation below:
Above,
 P(c|x) is the posterior probability of class (c, target) given predictor (x, attributes).
 P(c) is the prior probability of class.
 P(x|c) is the likelihood which is the probability of predictor given class.
 P(x) is the prior probability of predictor.
Dept. of CS&E, AJIET, Mangalore
32
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Program 6:
#Read and Handle Data
import csv
import random
import math
#random.seed(0)
# 1.Data Handling:
# 1.1 Loading the Data from csv file of Pima indians diabetes dataset.
def loadcsv(filename):
reader = csv.reader(open(filename, "r"))
dataset = [ ]
for row in reader:
inlist = [ ]
for i in range(len(row)):#converting the attributes from string to floating point numbers
inlist.append(float(row[i]))
dataset.append(inlist)
return dataset
#splitting the Data set into Training Set
#The naive bayes model is comprised of a summary of the data in the training dataset.
#This summary is then used when making predictions.
#involves the mean and the standard deviation for each attribute, by class value
def splitDataset(dataset, splitRatio):
trainSize = int( len(dataset) * splitRatio)
trainSet = []
copy = list(dataset)
while len(trainSet) < trainSize:
index = random.randrange(len(copy)) # random index
trainSet.append(copy.pop(index))
return [trainSet, copy]
#Summarize Data
#Separate Data by Class
#Function to categorize the dataset in terms of classes.
#The function assumes that the last attribute (-1) is the class value.
#The function returns a map of class values to lists of data instances.
def separateByClass(dataset):
separated = {}
for i in range( len(dataset) ):
vector = dataset[i]
if (vector[-1] not in separated):
separated[vector[-1]] = []
separated[vector[-1]].append(vector)
return separated
Dept. of CS&E, AJIET, Mangalore
33
Artificial Intelligence & Machine Learning Laboratory
18CSL76
#Calculate the Mean
#The mean is the central middle or central tendency of the data, and we will use it as the
#middle of our gaussian distribution when calculating probabilities
#2.2 : Calculate Mean
def mean(numbers):
return sum(numbers)/float(len(numbers))
#Calculate the Standard Deviation
#The standard deviation describes the variation of spread of the data, and we will use it to
#characterize the expected spread of each attribute in our Gaussian distribution when
#calculating probabilities.
#2.3 : Calculate Standard Deviation
def stdev(numbers):
if len(numbers)==0:
return 0
avg = mean(numbers)
variance = sum( [ pow(x-avg, 2) for x in numbers]) / float(len(numbers)-1)
return math.sqrt(variance)
#Summarize Dataset
def summarize(dataset):
summaries = [ (mean(attribute), stdev(attribute)) for attribute in zip(*dataset) ]
del summaries[-1]
return summaries
#Summarize Attributes By Class
def summarizeByClass(dataset):
separated = separateByClass(dataset)
#print(separated)
summaries = {}
for classValue, instances in separated.items():
summaries[classValue] = summarize(instances)
return summaries
#Make Prediction
#Calculate Probaility Density Function
def calculateProbability(x, mean, stdev):
exponent = math.exp( -(math.pow(x-mean,2) / (2*math.pow(stdev,2) ) ) )
return (1 / (math.sqrt(2*math.pi) * stdev)) * exponent
#Calculate Class Probabilities
def calculateClassProbabilities(summaries, inputVector):
probabilities = {}
Dept. of CS&E, AJIET, Mangalore
34
Artificial Intelligence & Machine Learning Laboratory
18CSL76
for classValue, classSummaries in summaries.items():
probabilities[classValue] = 1
for i in range(len(classSummaries)):
mean, stdev = classSummaries[i]
x = inputVector[i]
probabilities[classValue] *= calculateProbability(x, mean, stdev)
return probabilities
#Prediction : look for the largest probability and return the associated class
def predict(summaries, inputVector):
probabilities = calculateClassProbabilities(summaries, inputVector)
#print(probabilities)
bestLabel, bestProb = None, -1
for classValue, probability in probabilities.items():
if bestLabel is None or probability > bestProb:
bestProb = probability
bestLabel = classValue
return bestLabel
#Make Predictions
def getPredictions(summaries, testSet):
predictions = []
for i in range(len(testSet)):
result = predict(summaries, testSet[i])
predictions.append(result)
return predictions
#Computing Accuracy
def getAccuracy(testSet, predictions):
correct = 0
for i in range(len(testSet)):
if testSet[i][-1] == predictions[i]:
correct += 1
return (correct / float(len(testSet))) * 100.0
#Main Function
def main():
filename ='PI_Diabetes.csv'
splitRatio = 0.67
dataset = loadcsv(filename)
print("\n The length of the Data Set : ", len(dataset))
print("\n The Data Set Splitting into Training and Testing \n")
trainingSet, testSet = splitDataset(dataset, splitRatio)
print('\n Number of Rows in Training Set:{0} rows '.format(len(trainingSet)))
print('\n Number of Rows in Testing Set:{0} rows '.format(len(testSet)))
# prepare model
summaries = summarizeByClass(trainingSet)
Dept. of CS&E, AJIET, Mangalore
35
Artificial Intelligence & Machine Learning Laboratory
18CSL76
print("\n Model Summaries:\n",summaries)
# test model
predictions = getPredictions(summaries, testSet)
print("\nPredictions:\n",predictions)
accuracy = getAccuracy(testSet, predictions)
print('\n Accuracy: {0}%'.format(accuracy))
main()
Output Example
Data set
Output Obtained is as follows;
The length of the Data Set : 768
The Data Set Splitting into Training and Testing
Number of Rows in Training Set: 514 rows
Number of Rows in Testing Set: 254 rows
Dept. of CS&E, AJIET, Mangalore
36
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Model Summaries:
{0.0: [(3.2560240963855422, 2.8693809935989156), (111.0632530120482, 26.7357931151
23844), (68.23192771084338, 17.762204141676488), (21.126506024096386, 14.721450607
767649), (70.45180722891567, 95.61166150347556), (30.96536144578313, 7.04128774455
2887), (0.4245090361445784, 0.2910541574633692), (31.268072289156628, 11.574419858
64711)], 1.0: [(4.8901098901098905, 3.689467012819897), (137.77472527472528, 32.8618
205089383), (69.02197802197803, 22.789926058841655), (21.692307692307693, 18.16632
3227816612), (93.4010989010989, 128.45842717625771), (34.75549450549449, 6.8956025
32375257), (0.5505164835164835, 0.36756391897658064), (37.27472527472528, 10.99629
865810549)]}
Predictions:
[1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.
0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0,
1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0
.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0
, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0,
1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.
0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0,
1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0
.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0
, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0,
1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
Accuracy: 76.77165354330708%
Viva Questions:
1.
2.
3.
What is Naives Bayes Classifier?
Answer: It is a classification technique based on Bayes‟ Theorem with an assumption
of independence among predictors.
Mention any three applications of Naïve Bayes Classifier.
Answer:
1. Spam Classification
– Given an email, predict whether it is spam or not
2. Medical Diagnosis
– Given a list of symptoms, predict whether a patient has disease X or not
3. Weather
– Based on temperature, humidity, etc… predict if it will rain tomorrow
Define Bayesian Theorem.
Answer:
Dept. of CS&E, AJIET, Mangalore
37
Artificial Intelligence & Machine Learning Laboratory
18CSL76
PROGRAM 7:
Q.7. Apply EM algorithm to cluster a set of data stored in a .CSV file. Use the same
data set for clustering using k-Means algorithm. Compare the results of these two
algorithms and comment on the quality of clustering. You can add Java/Python
ML library classes/API in the program.
Theory
These are the two basic steps of the EM algorithm, namely E Step or Expectation Step or
Estimation Step and M Step or Maximization Step.
Estimation step:
• Initializ μk, Σk and πk by some random values, or by K means clustering results or by
hierarchical clustering results.
• Then for those given parameter values, estimate the value of the latent variables (i.e γk)
Maximization Step:
• Update the value of the parameters( i.e. μk, Σk and πk) calculated using ML method.
1. Select the number of seeds. Let this number be k
2. Pick k seeds as centroids of the k clusters. The seeds may be randomly picked.
3. Compute Euclidean/Manhattan distance of each object in the dataset from each of the
centroids.
4. Allocate each object to the cluster it is nearest to based on the distances computed in the
previous step.
5. Compute the centroids of the clusters by computing the means of the attribute values of
the objects in each cluster.
6. Check if the stopping criterion has been met. (ex: cluster membership is unchanged). If so
go to step 7. if not got to step 3
7. [optional] one may decide to stop at this stage or to split a cluster or combine two clusters
heauristically until a stopping criterion is met.
Algorithm:







Step 1: Given a set of incomplete data, start with a set of initialized parameters.
Step 2: Expectation step (E – step): In this expectation step, by using the observed
available data of the dataset, we can try to estimate or guess the values of the
missing data. Finally, after this step, we get complete data having no missing values.
Step 3: Maximization step (M – step): Now, we have to use the complete data, which is
prepared in the expectation step, and update the parameters.
Step 4: Repeat step 2 and step 3 until we converge to our solution.
Dept. of CS&E, AJIET, Mangalore
38
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Program 7:
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from sklearn import preprocessing
from sklearn.mixture import GaussianMixture
import sklearn.metrics as sm
import pandas as pd
import numpy as np #%matplotlib inline # Load the iris dataset
iris_dataset = pd.read_csv('iris.csv')
iris_dataset['Targets'] = iris_dataset.Class.map({'Iris-setosa':0, 'Iris-versicolor':1, 'Iris-virginica':2})
X = iris_dataset[['Sepal_Length', 'Sepal_Width', 'Petal_Length', 'Petal_Width' ] ]
Y = iris_dataset[['Targets']]
#print(Y)
#Build K-Means Model
model = KMeans(n_clusters = 3)
model.fit(X)
print('Model Labels:\n', model.labels_ )
#Build GMM for EM algorithm
scaler = preprocessing.StandardScaler()
scaler.fit(X)
xs = scaler.transform(X)
gmm = GaussianMixture(n_components = 3)
gmm.fit(xs)
Y_gmm = gmm.predict(xs)
print('GMM Labels:\n', Y_gmm) #Visualize the clustering results
# To View the results set the size of the plot
plt.figure(figsize = (10, 10))
# Create a colormap
colormap = np.array(['red', 'lime', 'black']) # Plot Orginal using Petal features
plt.subplot(2, 2, 1)
plt.scatter(X.Petal_Length,X.Petal_Width, c = colormap[Y.Targets], s=40)
plt.title('Real Classification')
plt.xlabel('Petal Length')
plt.ylabel('Petal Width')
# Plot the K-Means model classifications
plt.subplot(2, 2, 2)
plt.scatter(X.Petal_Length,X.Petal_Width, c = colormap[model.labels_], s=40)
plt.title('K Means Clustering')
plt.xlabel('Petal Length')
plt.ylabel('Petal Width')
#Plot the GMM Model classification
plt.subplot(2, 2, 3)
plt.scatter(X.Petal_Length, X.Petal_Width, c = colormap[Y_gmm], s=40)
plt.title('GMM Based Clustering')
plt.xlabel('Petal Length')
plt.ylabel('Petal Width')
#Calculate performance metrics for K-Means and GMM
Dept. of CS&E, AJIET, Mangalore
39
Artificial Intelligence & Machine Learning Laboratory
18CSL76
print('Evaluation of K-Means with ground truth classification of Iris Dataset')
print('Rand Index:%f ' % sm.adjusted_rand_score(Y.Targets, model.labels_ ))
print('Homogenity Score:%f ' % sm.homogeneity_score(Y.Targets, model.labels_ ))
print('Completeness Score:%f ' % sm.completeness_score(Y.Targets, model.labels_ ))
print('V-Measure:%f ' % sm.v_measure_score(Y.Targets, model.labels_ ))
print('Evaluation of GMM with ground truth classification of Iris Dataset')
print('Rand Index:%f ' % sm.adjusted_rand_score(Y.Targets, Y_gmm))
print('Homogenity Score:%f ' % sm.homogeneity_score(Y.Targets, Y_gmm))
print('Completeness Score:%f ' % sm.completeness_score(Y.Targets, Y_gmm))
print('V-Measure:%f ' % sm.v_measure_score(Y.Targets, Y_gmm))
Output Example
Model Labels:
[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1111111111111002000000000000000000000
0002000000000000000000000020222202222
2200222202020220022222022220222022202
2 0]
GMM Labels:
[2 2 2 1 2 2 2 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 1 2 2 2 2 2 2 2
2122122222222000000000000000000000000
0000000000000000000000000000000000000
0000000000000000000000000000000000000
0 0]
Evaluation of K-Means with ground truth classification of Iris Dataset
Rand Index:0.730238
Homogenity Score:0.751485
Completeness Score:0.764986
V-Measure:0.758176
Evaluation of GMM with ground truth classification of Iris Dataset
Rand Index:0.507349
Homogenity Score:0.579380
Completeness Score:0.825032
V-Measure:0.680722
Dept. of CS&E, AJIET, Mangalore
40
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Viva Questions:
1. What is EM algorithm?
Answer: Expectation Maximization (EM) algorithm. Since we do not have the values
for the not observed (latent) variables, the Expectation-Maximization algorithm tries
to use the existing data to determine the optimum values for these variables and then
finds the model parameters.
2. How many steps are performed in this EM algorithm and what are they?
Answer: There are basically two steps performed, and they are Expectation Step and
Maximization Step.
Dept. of CS&E, AJIET, Mangalore
41
Artificial Intelligence & Machine Learning Laboratory
18CSL76
PROGRAM 8
Q.8. Write a program to implement k-Nearest Neighbour algorithm to classify the iris
data set. Print both correct and wrong predictions. Java/Python ML Library classes
can be used for this problem.
Theory



K-Nearest Neighbors is one of the most basic yet essential classification algorithms in
Machine Learning. It belongs to the supervised learning domain and finds intense
application in pattern recognition, data mining and intrusion detection.
It is widely disposable in real-life scenarios since it is non-parametric, meaning, it
does not make any underlying assumptions about the distribution of data.
Algorithm
Input: Let m be the number of training data samples. Let p be an unknown point.
Method:
1. Store the training samples in an array of data points arr[]. This means each
element of this array represents a tuple (x, y).
2. for i=0 to m
Calculate Euclidean distance d(arr[i], p).
3. Make set S of K smallest distances obtained. Each of these distances
correspond to an already classified data point.
4. Return the majority label among S.
Algorithm:
Step-1: Select the number K of the neighbors
Step-2: Calculate the Euclidean distance of K number of neighbors
Step-3: Take the K nearest neighbors as per the calculated Euclidean distance.
Step-4: Among these k neighbors, count the number of the data points in each category.
Step-5: Assign the new data points to that category for which the number of the neighbor is
maximum.
Step-6: Our model is ready.
Program 8:
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn import datasets
iris=datasets.load_iris()
print("Iris Data set loaded...")
#print(iris.data)
#print(iris.target)
Dept. of CS&E, AJIET, Mangalore
42
Artificial Intelligence & Machine Learning Laboratory
18CSL76
x_train, x_test, y_train, y_test = train_test_split(iris.data,iris.target,test_size=0.1)
#random_state=0
print("Dataset is split into training and testing samples...")
print("Size of trainng data and its label",x_train.shape,y_train.shape)
print("Size of testing data and its label",x_test.shape, y_test.shape)
for i in range(len(iris.target_names)):
print("Label", i , "-",str(iris.target_names[i]))
classifier = KNeighborsClassifier(n_neighbors=1)
classifier.fit(x_train, y_train)
y_pred=classifier.predict(x_test)
print("Results of Classification using K-nn with K=1 ")
for r in range(0,len(x_test)):
print(" Sample:", str(x_test[r]), " Actual-label:", str(y_test[r])," Predicted-label:", str(y_pred[r]))
print("Classification Accuracy :" , classifier.score(x_test,y_test));
Output Example
Iris Data set loaded...
Dataset is split into training and testing samples...
Size of trainng data and its label (135, 4) (135,)
Size of testing data and its label (15, 4) (15,)
Label 0 - setosa
Label 1 - versicolor
Label 2 - virginica
Results of Classification using K-nn with K=1
Sample: [5.9 3. 5.1 1.8] Actual-label: 2 Predicted-label: 2
Sample: [6.1 3. 4.9 1.8] Actual-label: 2 Predicted-label: 2
Sample: [6.9 3.1 4.9 1.5] Actual-label: 1 Predicted-label: 1
Sample: [5.5 2.5 4. 1.3] Actual-label: 1 Predicted-label: 1
Sample: [6. 2.2 5. 1.5] Actual-label: 2 Predicted-label: 1
Sample: [5.1 3.4 1.5 0.2] Actual-label: 0 Predicted-label: 0
Sample: [7.2 3.2 6. 1.8] Actual-label: 2 Predicted-label: 2
Sample: [5.5 4.2 1.4 0.2] Actual-label: 0 Predicted-label: 0
Sample: [6.3 3.3 6. 2.5] Actual-label: 2 Predicted-label: 2
Sample: [6. 2.7 5.1 1.6] Actual-label: 1 Predicted-label: 2
Sample: [6.7 3.3 5.7 2.5] Actual-label: 2 Predicted-label: 2
Sample: [4.9 3.1 1.5 0.1] Actual-label: 0 Predicted-label: 0
Sample: [4.7 3.2 1.3 0.2] Actual-label: 0 Predicted-label: 0
Sample: [6.4 3.2 4.5 1.5] Actual-label: 1 Predicted-label: 1
Sample: [5.7 4.4 1.5 0.4] Actual-label: 0 Predicted-label: 0
Classification Accuracy : 0.8666666666666667
Viva Questions:
1. Is KNN supervised or un-supervised learning algorithm?
Answer: KNN is a supervised learning algorithm
2. Name any application of KNN algorithm.
Answer: Can be used for Recommendation Sytems. Example: Recommendation is
Netflix, Amazon for the customers.
Dept. of CS&E, AJIET, Mangalore
43
Artificial Intelligence & Machine Learning Laboratory
18CSL76
PROGRAM 9
Q.9. Implement the non-parametric Locally Weighted Regression algorithm in order to
fit data points. Select appropriate data set for your experiment and draw graphs.
Theory
Regression:
• Regression is a technique from statistics that is used to predict values of a desired
target quantity when the target quantity is continuous .
• In regression, we seek to identify (or estimate) a continuous variable y associated
with a given input vector x.
y is called the dependent variable.
x is called the independent variable.
Loess/Lowess Regression: Loess regression is a nonparametric technique that uses local
weighted regression to fit a smooth curve through points in a scatter plot.
Algorithm:
Step 1: Read the Given data Sample to X and the curve (linear or non linear) to Y
Step 2: Set the value for Smoothening parameter or Free parameter say τ
Step 3: Set the bias /Point of interest set X0 which is a subset of X
Step 4: Determine the weight matrix using :
Step 5: Determine the value of model term parameter β using :
Step 6: Prediction = x0*β
Dept. of CS&E, AJIET, Mangalore
44
Artificial Intelligence & Machine Learning Laboratory
18CSL76
Program 9:
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
def kernel(point, xmat, k):
m, n = np.shape(xmat)
weights = np.mat(np.eye((m))) # eye - identity matrix
for j in range(m):
diff = point - X[j]
weights[j, j] = np.exp(diff*diff.T/(-2.0*k**2))
return weights
def localWeight(point, xmat, ymat, k):
wei = kernel(point,xmat,k)
W = (X.T*(wei*X)).I*(X.T*(wei*ymat.T))
return W
def localWeightRegression(xmat, ymat, k):
m, n = np.shape(xmat)
#print(m)
#print(n)
ypred = np.zeros(m)
for i in range(m):
ypred[i] = xmat[i]*localWeight(xmat[i], xmat, ymat, k)
return ypred
def graphPlot(X, ypred):
sortindex = X[:,1].argsort(0) #argsort - index of the smallest
xsort = X[sortindex][:,0]
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(bill,tip, color='green')
ax.plot(xsort[:,1],ypred[sortindex], color = 'red', linewidth=5)
plt.xlabel('Total bill')
plt.ylabel('Tip')
plt.show()
# load data points
data = pd.read_csv('Tips.csv')
bill = np.array(data.total_bill) # We use only Bill amount and Tips data
tip = np.array(data.tip)
mbill = np.mat(bill) # .mat will convert nd array is converted in 2D array
mtip = np.mat(tip)
#print(mtip)
m= np.shape(mbill)[1]
#print(m)
one = np.mat(np.ones(m))
X = np.hstack((one.T, mbill.T)) # 244 rows, 2 cols
Dept. of CS&E, AJIET, Mangalore
45
Artificial Intelligence & Machine Learning Laboratory
18CSL76
#print(X)
# Prediction with k=3
print('\nypred for k=3')
ypred = localWeightRegression(X,mtip,3)
graphPlot(X, ypred)
# Prediction with k=9
print('\nypred for k=9')
ypred = localWeightRegression(X, mtip, 9)
graphPlot(X, ypred)
Output Example
ypred for k=3
ypred for k=9
Viva Questions:
1. What is Regression?
Answer: Regression is a technique from statistics that is used to predict values of a
desired target quantity when the target quantity is continuous .
2. Why do we use Regression?
Answer: In regression, we seek to identify (or estimate) a continuous variable y
associated with a given input vector x.
Dept. of CS&E, AJIET, Mangalore
46