Hieu Le
Novelty search – a promising approach for constructing Ensemble system.
Introduction
In the field of Artificial Intelligence, Classification problem is one of the most popular problems
with wide applications in various fields including healthcare and agriculture. One of the classic examples
is probably disease detection. Given a report with set of values regarding a patient health (blood pressure,
height, blood type etc.) detect whether the patient has a particular disease. More formally, a classification
problem is given a set of values regarding some particular fields; determine whether the set of values
(input) belongs to a particular class. The set of values is often denoted in vector form as <x1, x2, x3, … xn>
where xi is a value of corresponding criteria (height, weight etc.). The classifier is trained beforehand with
a set of training examples and returns a hypothesis/class about the input vector.
Improving accuracy of the classifier is an interesting task which attracts a lot of researchers. One
well-known approach is Ensemble methods. According to Dietterich (2000), Ensemble methods conduct a
hypothesis based on average vote from an ensemble/set of classifiers. The idea behind ensemble methods
is quite intuitive: an ensemble with many classifiers may have higher possibility of giving a correct
hypothesis than a single classifier. Even when one classifier returns a wrong hypothesis, the ensemble
may still give correct hypothesis if the majority of classifiers work correctly. However, there is also a
chance that the majority if not all of the classifiers make the same mistakes, which make the ensemble
methods fall short. This give the notion that classifiers should behaves differently to avoid the case which
many classifiers make the same mistakes.
Since we want an ensemble systems consisted of classifiers with different behaviors, an
interesting question is how to construct such ensemble system. In other words, how should we pick
classifiers such that they can form an ensemble with a very broad range of classifiers’ behaviors? This
question will be my main research topic for the honor project under the guidance of professor Braught.
Some methods have been applied to construct the desired ensemble systems such as bagging or
ADABoost (Dietterich, 2000). On the other hand, the proposed method for this research question will be
novelty search, which will be described in the next section. The goal of the research will be applying
novelty search to the ensemble construction process and compared the result with other methods used to
construct ensemble system.
Method for research.
Novelty search is a new approach to Evolutionary computation. The intuitive and widely applied
approach is evolving toward the goal of the search space (Lehman, 2011). More specifically, with some
possible next steps, the system will choose the step that is closest to the goal using an objective function.
However, the evolving with objective approach can still fail if objective function deceptively chooses a
step that seems closer to the goal but actually leads to local optima. Novelty search effectively avoids
local optima because it doesn’t explicitly look for an object but instead look for behavioral novelty
(Lehman, 2000). Novelty search looks for new behaviors by keeping a collection of known behaviors and
comparing new behaviors with the known ones to find out the most radical behavior. According to
Lehman (2000), a simple novelty metric is calculating the “average distance to the k-nearest neighbors”
of a given point. The further one point is from other known behaviors, the more novel it is. This approach
may seem less effective compared to objective search but experiments on problems with high level of
deception (more likely to fall into local optima) have shown that novelty search can outperform objective
search in some cases.
With the promising result from other researches, this research project will attempt to apply
novelty search to classifiers space to select classifiers that behave differently to construct ensemble
system.
References
Dietterich, T. G. (2000). Ensemble Methods in Machine Learning. In J. Kittler and F. Roli (Ed.) First
International Workshop on Multiple Classifier Systems, Lecture Notes in Computer Science (pp.
1-15). New York: Springer Verlag.
Lehman, J. Stanley, K. (2011). Abandoning Objectives: Evolution through the Search for Novelty Alone.
Evolutionary Computation (19):2, pages 189-223, Cambridge, MA: MIT Press
Lehman, J. Stanley, K. (2010). Revising the Evolutionary Computation Abstraction: Minimal Criteria
Novelty Search. Proceedings of the Genetic and Evolutionary Computation Conference
(GECCO-2010). New York, NY:ACM
Opitz, D. Shavlik, J (1996). Actively Searching for an Effective Neural-Network Ensemble. Connection
Science, 8, pp. 337-353.
Sallam, H, Regazzoni, C. (2008). Evolving neural networks ensembles NNEs. Cognitive Information
Processing.