Learning from Hamming
Why the Big Problems Matter
Peter J Bentley
Google: “you and your research hamming”
Richard Hamming
•
•
•
•
•
Hamming distance
Hamming code
Hamming matrix
Hamming window
Hamming bound
Richard Hamming: you and your research
•
I saw Feynman up close. I saw Fermi and Teller. I saw
Oppenheimer. I saw Hans Bethe: he was my boss. I saw
quite a few very capable people. I shared an office for a while
with Shannon.
•
I became very interested in the difference between those who
do and those who might have done.
•
One of the characteristics of successful scientists is having
courage. Once you get your courage up and believe that you
can do important problems, then you can. If you think you
can't, almost surely you are not going to.
Richard Hamming: you and your research
•
I went over and said, ``Do you mind if I join you?'' They can't
say no, so I started eating with them for a while.
•
I started asking, ``What are the important problems of your
field?’’
•
And after a week or so, ``What important problems are you
working on?’’
•
And after some more time I came in one day and said, ``If
what you are doing is not important, and if you don't think it is
going to lead to something important, why are you working on
it?’’
•
I wasn't welcomed after that; I had to find somebody else to
eat with!
Search Based Software Engineering
• ``What are the important problems of your field?’’
Search Based Software Engineering
•
“SBSE could become a coherent field of activity that combines
the expertise and skills of the Metaheuristic Search community
with those of the Software Engineering community.”
•
“Open problems: Stopping Criteria, Landscape Visualisation,
Characterizing the Software Engineering Search Spaces,
Human Competitive Results”
•
“Future developments: Multi Objective Optimization,
Interactive Optimization, Sensitivity Analysis, Hybrid
Optimization Algorithms, On Line Optimization”
•
“Benefits: Scalability, Robustness, Feedback and Insight”
Richard Hamming: you and your research
• ``What important problems are you working on?’’
Search Based Software Engineering
•
“Testing, Multiobjective optimisation, RE, program slicing,
constraint satisfaction”
•
BUT for multiobjective optimisation:
•
“Once the real world scenario has more than about 10
stakeholders, the visualisations we propose and the
animations we introduce will become hard to read.
Furthermore, the search-based optimisation techniques we
use do not, presently, scale well beyond four or five
objectives.
Example big(ish) problem:
Scalable Multi-objective Optimisation
•
•
The cause?
An unhealthy obsession with the “Pareto Optimal Front”:
Example big(ish) problem:
Scalable Multi-objective Optimisation
• Why not address this problem?
• Let’s do it now.
Pareto
•
Vilfredo Pareto 1848-1923
•
Italian engineer, sociologist, economist, and philosopher.
•
Pareto's biographer: "In the first years of his rule
Mussolini literally executed the policy prescribed by
Pareto, destroying political liberalism, but at the same
time largely replacing state management of private
enterprise, diminishing taxes on property, favoring
industrial development, imposing a religious education in
dogmas”.
•
Karl Popper called him the "Theoretician of
Totalitarianism"
Pareto’s words
Pareto, Vilfredo (1935). The Mind and Society [Trattato Di Sociologia Generale]. Harcourt, Brace.
•
Order is "the disposition of a number of things with reference to
some specified end and proportioned to a desired effect.”
•
"Propriety {convenance) approximates order itself. It is a
relationship of conformity [What is this conformity?] among
several things, each of which is in itself promotive of the
preservation and perfection of the other [And what this
perfection?], and does its share in maintaining it in a good and
advantageous estate.”
•
Good for whom? Advantageous for whom ? A poison that leaves
no trace "is promotive of the preservation and perfection" of the
man who wants to murder a neighbour, and maintains him in an
estate that is "good and advantageous" for him; but it cannot be
said to be "promotive of the preservation and perfection" of the
victim, or that it maintains the victim in a "good and
advantageous estate."
Pareto optimality is from economics
•
Vijay K. Mathur, "How Well Do We Know Pareto Optimality?" Journal of
Economic Education 22:2 (1991) pp 172-78
•
“Pareto realized that it was not necessary
to know how much a person valued this
or that, only that he preferred X of this to
Y of that. Pareto said ‘good’ cannot be
measured.”
•
“He replaced it with the notion of Paretooptimality, the idea that a system is
enjoying maximum economic satisfaction
when no one can be made better off
without making someone else worse off.”
Pareto optimality and genetic algorithms
•
Multiobjective problem solving from nature: from concepts to
applications by Joshua Knowles, David Corne, Kalyanmoy Deb
“At the end you have a set of solutions - a set of bridge
designs, or factory production schedules, or whatever. So,
why do multiobjective at all? Well the difference is potentially
twofold. first, via the multiobjective search you may have found a better result, in
terms of the single-objective score… Second you are presented with a diverse
set of solutions that provides information about the trade offs available to you.”
•
“True what we are suggesting here is that a decision needs to be made and in
that sense the multiobjective search seems not to have automatically solved
your problem for you.”
•
“Multiobjective search is therefore viewed as a way of providing the opportunity
for a decision maker to make informed decisions about the solution based on the
information about the solutions that inhabit the Pareto front.”
Pareto optimality and genetic algorithms
•
But it doesn’t scale!
•
For many objectives, forced
diversity of populations and a
weak selection pressure onto
a messy and disparate Pareto
front results in no convergence
at all, and failure.
Pareto optimality and genetic algorithms
•
Two objectives is not multi-objective. Twenty is
multi-objective!
•
In real world problems, those objectives may be
unknown black boxes that continuously change
over time, are noisy, discontinuous, multimodal
and constrained.
There are no perfect “knee points” for real world
problems.
Clustering objectives just throws away
information.
You can’t visualise Pareto fronts in these cases.
The whole idea doesn’t work!
And do clients actually want to look at Pareto
fronts anyway? Would they not prefer to just get
the optimum solution?
•
•
•
•
Pareto optimality and genetic algorithms
•
So if Pareto optimality doesn’t work, why not just look for real optimality?
•
In the general case, you cannot say which answer is most optimal because in
the general case you do not know the preferences of the user.
•
But when we optimise, we do not seek an answer in general, we seek one
specific optimal that meets the preferences of the user.
•
We can learn these preferences (perhaps through interaction with the client),
use them, and thus optimise. We can judge which is best.
Pareto optimality and genetic algorithms
Bentley, PJ and Wakefield, JP (1998) Finding Acceptable Solutions in the Pareto-Optimal
Range using Multiobjective Genetic Algorithms. In: Chawdhry, PK and Roy, R and Pant, RK,
(eds.) Soft Computing in Engineering Design and Manufacturing. (pp. 231 - 240). SpringerVerlag: Berlin/Heidelberg, Germany.
(Work done in 1995)
Pareto optimality and genetic algorithms
•
Techniques for Highly Multiobjective Optimisation: Some Nondominated Points
are Better than Others by David Corne, Joshua Knowles. GECCO 2007.
•
“Simple variants of the often-overlooked ‘Average Ranking’ strategy usually outperform
other methods tested, covering problems with 5—20 objectives and differing amounts of
inter-objective correlation.”
•
“Our main finding is that ARF, based on the somewhat overlooked AR method [3] seems a
very strong candidate for many-objective search. It seems to outperform two recent
methods that have been proposed specifically for many-objective problems. Usefully, it is
also computationally simple and efficient.”
•
“Perhaps because it was not published in mainstream MOEA literature, and also perhaps
because the comparisons were not based on quality measures of the Pareto set, and also
perhaps because it tends to be associated with objective-priority approaches, Bentley &
Wakefield’s WAR has been little used, omitted for example in the several large or
moderate scale MOEA comparison papers.”
•
“From our results, we recommend, of course, that ARF be further tested and used in
many- objective scenarios; but we also suggest both ARF and AR be revisited as
candidates for ‘standard’ (2—5 objective) problems; preliminary studies indicate that it
performs comparatively well in such cases too.”
Richard Hamming: you and your research
• And after some more time I came in one day and said,
``If what you are doing is not important, and if you don't
think it is going to lead to something important, why are
you working on it?’’
• I wasn't welcomed after that; I had to find somebody
else to eat with!
SBSE is young. Don’t let it be like this:
• Simply combining two successful areas (optimisation and SE) does
not equal success. You need to optimise the ideas themselves and
form a new, cohesive whole.
Some Big Dangers in SBSE
• Don’t use or reinvent the same flawed wheel when it looks like this:
•
(multiobjective optimisation, constraint handling, scalable search)
Some Big Dangers in SBSE
• Beware! When you’re holding a
hammer, everything looks like a
nail.
• For example, if you can calculate
the answer directly or other
methods solve the problem better
(e.g. best prioritised list of
requirements), search is the wrong
tool for the job.
Some Big Problems in SBSE
• Learn which search methods should be used with which SE problems.
• Develop new search methods more suitable for SE problems.
• Develop new SE methods compatible with search.
• Surely SBSE should exemplify cutting edge work in search and
SE, not gluing old or inappropriate ideas together to make flying
cars.
Search & Software Engineering
• ``What are the important problems of your field?’’
Big problems in Search
•
Scalabilty
•
Complexity
•
Evolvability
•
Adaptive systems
•
Coevolving systems
•
Good representation design
•
And many others…
Big problems in SE
•
“The major challenge faced by software projects is
increasing complexity as we create systems by integrating
systems and services from different providers.”
•
“The assumption that underlies much of software
engineering is that the developer of the software is in
control of the system and can therefore make rational
decisions about how to develop and change the system.”
•
“When this assumption is no longer true, methods such as
software testing methods have to evolve to take this into
account.”
Big problems in SE
•
“Engineering scalability: ‘Internet-scale’ services.
Handling large and rapid variations in the demand for
resources”
•
“Resource estimation. We are unable to reliably predict
the cost/effort required to build a system.”
•
Development of emerging classes of ‘adaptive’ system
•
The apotheosis of ‘apps’. Requirements engineering
for mass-markets?
•
Software project failures. Mismatches at the
boundaries between business and software
engineering?
•
Addressing complex inter-product and inter-supplier
dependencies. Software ecosystems?
Richard Hamming: you and your research
• When you get beyond their modesty, most people will
say, ``Yes, doing really first-class work, and knowing it,
is as good as wine, women and song put together,'' or if
it's a woman she says, ``It is as good as wine, men and
song put together.''
• I think it is very definitely worth the struggle to try and
do first-class work because the truth is, the value is in
the struggle more than it is in the result. The struggle to
make something of yourself seems to be worthwhile in
itself. The success and fame are sort of dividends, in
my opinion.
• Go forth, then, and do great work!
Questions?