5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
Group 5:
Historical control data
• Follow-up of 2005 IWGT where the use of historical control
data in interpretation of in vitro results was identified as an
important topic, but was not discussed in depth because of
lack of time.
• A short session took place in Basel to further discussed
this topic.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
Assigned Participants
Kerry Dearfield
Makoto Hayashi (Chair)
David Jacobson-Kram: not able to attend
Peter Kasper
David Lovell (Co-chair)
Hans-Joerg Martus
Veronique Thybaud (Rapporteur)
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
Objectives
• The group mainly discussed negative control data.
• Historical control data:
- Until now used for acceptance criteria.
- More and more used also for interpretation of results.
• Up to now there is no clear guidance.
• Objectives of the group are:
- To provide generic advice for historical control data that could
be applied to all assays, not to describe assay specific
recommendations.
- To make consensus on the following questions:
What are the historical control data?
 How to make the historical control data?
 How to use the historical control data?

• Quite helpful, formed the
basis of initial discussion
• Statements still
acceptable, but needed to
be widened to include
full range of assays
including in vitro ones.
An example of use for in vivo
assays, i.e. data distribution
and the use of quality control
charts to explore changes
over in negative control data
in a large set of in vivo MNT
data generated in a same
laboratory.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
What are the historical control data? -1• It is the complete set of data from previous experiments.
• It includes both individual values and the means of the
negative control groups of that experiment.
• Individual values:
For in vivo assays: animals
For in vitro assays: cultures, plates
• All experiments must have a concurrent negative control
and all concurrent negative controls should be added to
derive at a representative negative historical control.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to build up the historical control data? -1• When you are building up historical control data you put in all
data that have been accumulated over the past.
• Ideally data should be accumulated in order to visualize the
distribution for individuals and the group means, and to
calculate the key parameters to be used for data analysis.
• Graphical control charts may be useful for exploring the data.
• Data should only be excluded from the data set when there is
a clear and valid reason to reject it.
• Reporting only a range (i.e., minimum, maximum) of data
from the historical studies is not adequate.
• The data set should be managed so that more recent data
carry more weight that older data.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to build up the historical control data? -2• The choice of the specific statistics such as mean, median etc.
for the end-points to be assessed in historical control data
should be determined by the assay community, based upon
their experience on the most appropriate parameters.
• Historical control data set should ideally be as big as
possible but probably comprise at least 10 separate
experiments, but preferably 20.
• However, this may depend upon the use of historical data
(e.g., acceptance criteria, distribution, internal quality control),
on the variability within an assay, and also on the
practicability (number of assays per year, etc).
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to build up the historical control data? -3• The experimental protocol must have remained fixed
throughout the period covered by the historical data and
the current experiment, unless it can be demonstrated that
the change does not impact the values.
- The method of scoring must be unchanged during the relevant period.
- The experimental units must be comparable throughout the period.
- There must exist no known systematic differences between the various
control groups, current and historical, that would produce systematic
differences in response, e.g., the data must have been gathered by the
same investigators within the same laboratory.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to build up the historical control data? -4Outliers: May need more work.
• Criteria for exclusion:
- Should not be excluded just because unusual.
- Should only be excluded if scientifically justified, as could be part of
assay variability.
• If decisions are only based on “ranges”, outliers have
disproportionate effect, that is one reason why the use of ranges is
not recommended, and confidence interval type approaches are
preferred.
• Values should be consistent with data reported in literature.
• Acceptable “ranges” should be defined for each assay by expert
group, e.g., as already done for MLA.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to use the historical control data? -1• Two distinct approaches to addressing negative historical
control data:
- To assess whether the concurrent control data is consistent
with previous negative control data collected by that laboratory.
Each assay will have its own specific acceptance criteria.
- To assess within a laboratory whether the techniques
used are still in “control” based upon a series of negative
controls results. This can be assessed by formal quality control.
• The historical control is very important for the technical
establishment and familiarization of the assay at the
laboratory.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to use the historical control data? -2• Concurrent negative controls are essential:
- As part of the acceptance criteria for the evaluation of the
technical acceptance of the study.
- For appropriate statistical treatment of the data.
- To accept a test (by comparing concurrent negative control
values to the historical negative control).
- To help assess the biological relevance of experimental
data (e.g., statistically significant increases detected, but
values comparable to historical control data).
• Nevertheless, it can be considered as a two step process:
- First compare the experimental data with the concurrent negative
control,
- then consider if meaningful as compared to historical data
distribution.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to use the historical control data? -3To design an assay (1):
• Assay/studies should be designed (number of duplicates, number
of cells scored, etc.):
- To detect a biologically important increase over the background.
- To detect a predetermined effect size/ difference from background
(test specific) of the observed endpoint at an a-level of 0.05 and a
power of 80% was considered to be an easily achievable minimum
standard.
- Based on the a-level (type I error) for recognizing a predetermined
increment over the spontaneous level with an acceptable statistical
power (1-type II error).
• In some assays doubling is used but this may not mean the same
thing biologically for all assays and depends on the background.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to use the historical control data? -4To design an assay (2):
• It should be generally required that a clear description of the
design of the experiment including the power of the test, the
statistical methods used to analyze the data and the level of
significance actually achieved is given in every data report.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to use historical control data? -5•
How do you reject a control group, and therefore the experiment,
from an experiment because it is unacceptable:
- The group advice against the use of range for this acceptance
limit criteria (see earlier).
- These could be based upon either the observed spread of data or
on calculated from the distribution values using (say) 2 or 3 standard
deviations or upper 99% percentile from the mean of that which was
actually observed.
- The choice of criteria has to be determined for the individual
assay based upon the specific characteristics of the assay. Each assay
determines its own acceptance/limit criteria which is consistent with
historical data generated by that scientific community. For example,
could be useful to define performance standard and acceptable “range”
or interval confidence for all assays as done before for MLA, or 5% for
CA.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to use of historical control data? -6•
•
If a negative control group in an experiment is outside the
acceptance criteria the experiment is generally no longer valid.
How far outside the acceptance criteria is still valid should be
managed for each assay and should be consistent with values
reported in scientific community.
• However some use may be made with the information of the
experiment if for instance the treated group show a clear increase.
But care is needed in case such an increase is also a result of
artifact such as problem with the culture in an in vitro study that
could have impacted all cultures and the whole experiment.
• In case of negative results you would have to repeat the
experiment.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
How to use of historical control data? -7•
Historical control data may have a role in defining the size of
effect that is considered biologically important.
•
Such considerations may have a role in the future in putting
dose-response relationship seen in experimental studies into
perspective in the context of the identification of “practical or
pragmatic threshold”.
•
Considerable care would be needed in the choice of data to use
in such assessment and more work is probably needed to
assess the potential of this approach.
5th IWGT, Basel
August 17-19, 2009
Group 5: Historical control data
Thank you