Guidelines for Reporting Meta-Heuristics in LOG 741
In LOG 741 there are three assignments. These assignments follow the same
framework. You have been given a black box, and should answer some simple
questions about it, based on running some tests.
About Testing
Computational testing can be a difficult and time-consuming task. There are many
different dimensions to test, and usually the combined number of possibilities is
too high for complete testing. Usually there are a number of test instances, having
similar and/or different characteristics.
The basic question one wants answered is usually:
 Which method is best?
 Which parameter values are the best for my optimization method?
 What is the performance profile over time for my method?
o Is it worth the effort to search for a longer time?
Test instances. In our tests we use randomly generated instances of 0/1 MCKP
(Multi-Constrained Knapsack Problems).They can be divided into 27 distinct
groups, with 10 instances in each group. The distinguishing instance dimensions
are:
Number of Variables (100, 250, 500)
Number of Constraints (5, 10, 30)
Knapsack Density (25%, 50%, 75%) – this is the expected number of items
needed to fill the knapsack as a percentage of variables.
Testing. Some times only 1 test instance is available. At other times millions are
available. When testing, often the basic underlying question is: Is my method better
than the competition? It is important to define what is meant by better. Some
typical examples:
 Better in one test on a specific instance
 Better in 30 tests (with different starting points) on a specific instance.
 Better in 30 tests (with different starting points) on a class of instances
 Better in 30 tests (with different starting points) on all instances
Either compare values from the same instance, or use the average of a statistically
significant number of similar instances.
Some final points




Formulate some hypothesis to prove (or disprove)
Be systematic
Make fair comparisons
Remember to test also for long times, 100.000.000 iterations? 1 hour?
Answer the questions. This is often forgotten.
Justify the answers. This means that you should explain why you reach a specific
conclusion.
Statistical significance. Local Search Heuristic choices are very often based on
some generated random number. A single computational test can thus have a high
variance in the results produced (i.e. best solution found) when repeated with
different random seeds. To produce higher statistical significance, a single test may
be repeated repeatedly, as long as different random number sequences are used.
You should consult your statistics course literature, but mainly there are two
choices for statistical tests. One is to use a non-parametric test like Mann-Whitney,
or have enough samples to approach a normal distribution, and then use a
parametric test for normal distributions.
Be able to reproduce the results. There is often a need to rerun all the tests made.
This might have different reasons, most often it is because an error in the code is
detected.The following points can help this re-testing.
 Collect all tests in a set of batch-files. A good example is given in
assignment 1.
 If a random number generator is used, reproducibility of single tests
can be guaranteed by using the same random number seed, starting the
random number generator at the same place in its pseudo-random
sequence.
About the document










Only deliver 1 document
Length 3 – 10 pages
Add page numbers
The front page of the document should contain the name of the authors
Do not hand in log-files
Use e.g. EXCEL to make graphics, and then cut and paste into the report
Related text and graphics and tables should be together
Always reference a table or graphic.
Make the document self-contained
Make the document look nice.
o Pay attention to layout.
o Remember that someone else might read it
o A nice document puts the reader (grader) in a better mood.
Other
Max group size = 2