Searching for structure
in random field data
Keith J. Worsley12,
Thomas W. Yee3, Russell B. Millar3
1Department
of Mathematics and Statistics, McGill University,
2McConnell Brain Imaging Centre, Montreal Neurological Institute,
Montreal, Canada, and
3Department of Statistics, University of Auckland, New Zealand
www.math.mcgill.ca/keith
What is Data Mining?
The June 26, 2000, issue of TIME predicted that
one of the 10 hottest jobs of the 21st century
will be Data Mining:
“ … research gurus will be on hand to extract
useful tidbits from mountains of data,
pinpointing behaviour patterns for marketers
and epidemiologists alike.”
Some definitions:
• Data mining is the process of selecting, exploring, and modeling large
amounts of data to uncover previously unknown patterns for business
advantage (SAS 1998 Annual Report, p51)
• Data mining is the nontrivial process of identifying valid, novel, potentially
useful, and ultimately understandable patterns in data (Fayyad)
• Data mining is the process of discovering advantageous patterns in data
(John)
• Data mining is the computer automated exploratory data analysis of
(usually) large complex data sets (Freidman, 1998)
• Data mining is the search for valuable information in large volumes of data
(Weiss and Indurkhya, 1998)
• In contrast, Statistics is the science of collecting, organizing and presenting
data.
Why is it called “Data Mining”?
• Plentiful data can be mined for nuggets of gold (i.e. truth
/insight/knowledge) by sifting through vast amounts of raw data.
• Some statisticians have criticized it as “data dredging” or a
“fishing expedition” in the search of publishable P-values, or
“torturing the data until it confesses”.
• Many DM methods are heuristic, complex, computer intensive,
so their statistical properties are usually not tractable.
• The focus of DM is often prediction and not statistical inference.
• I understand mining to be a very carefully planned search for
valuables hidden out of sight, not a haphazard ramble. Mining is
thus rewarding, but, of course, a dangerous activity. (D.R. Cox,
in the discussion of Chatfield, 1995).
Striking fool’s gold
• “The Bible Code”, a best-selling book by Michael
Drosnin, claims to find hidden messages in the Bible
about dinosaurs, Bill Clinton, the Rabin assassination
etc. from searches of arrays of letters …
• In 1992, ProCyte Corp. was dismayed when a newly
developed drug, lamin, failed to promote general
healing of diabetic ulcer wounds. So the company
searched through subsets of data and found that lamin
appeared to work on certain foot wounds. But that
was a statistical fluke, as it turned out after an
expensive clinical trial. Not allowed drug status,
lamin is now sold as a wound dressing …
Confirming vs. Discovering
There are two types of DM:
1. Hypothesis testing (aka top-down approach)
2. Knowledge Discovery in Databases (KDD)
(aka bottom-up approach)
• Directed KDD: want to explain the value of some
particular variable in terms of other variables
• Undirected KDD: identifies patterns in the data.
Undirected KDD recognizes relationships in data;
Directed KDD explains those relationships once they
have been found.
Mining the miners
DM so far has been largely a commercial enterprise. As
in most gold rushes of the past, the goal is to “mine
the miners”. The largest profits are made by selling
the tools to the miners, rather than in doing the actual
mining:
• Hardware manufacturers emphasize high
computational requirements of DM.
• Software developers emphasize competitive edge
“Your competitor is doing it, so you had better keep
up.”
Some commercial software
• SAS “Enterprise Miner”
• SPSS “Clementine”, “Neural Connection”
and “AnswerTree”
• IBM “Intelligent Miner”
• SGI “MineSet”
• NeoVista Software “ASIC”
• Mathsoft “S-PLUS” (for small data sets)
Some methods
• Hypothesis testing: Regression, analysis of
variance, time series analysis.
• Directed KDD: Classification, discrimination,
structural equation modeling, supervised
neural networks.
• Undirected KDD: Cluster analysis, tree
methods (AID, CHAID, CART), principal
components analysis (PCA), independent
components analysis (ICA), unsupervised
neural networks.
Allied fields
• Exploratory Data Analysis (EDA): Tukey defined
statistics in terms of problems rather than tools.
• Informatics “is research on, development of, and use of technological, sociological, and
organizational tools and applications for the dynamic acquisition, indexing, dissemination, storage,
querying, retrieval, visualization, integration, analysis, synthesis, sharing (which includes electronic means
of collaboration), and publication of data such that economic and other benefits may be derived from the
information by users of all sections of society.”
• Pattern recognition: given some examples of complex signals
and the correct decisions for them, make decisions
automatically for a stream of future examples, e.g. identify
plants, tumors, decide to buy or sell stocks.
• Machine learning “is the study of computer algorithms that improve
automatically through experience. Applications range from data mining programs
that discover rules in large data sets, to information filtering systems that
automatically learn users’ interests.” (Mitchell, 1997).
• Meta-Analysis is the statistical analysis of a large collection of
analysis results from individual studies for the purpose of
integrating the findings.
Brain mapping data
• We have huge data bases of brain images (MRI,
fMRI, PET, EEG, MEG …) together with patient
information (age, sex, psychological tests, disease,
genotype …)
• The novelty is that the image variables are 3D images
rather than single numbers (such as blood pressure,
cholesterol level …)
• These images can themselves be mined for interesting
information, e.g. peaks or clusters of activated
regions
Some data mining tools already
used in brain mapping
• Regression, analysis of variance, time series
• Cluster analysis (e.g. clustering of fMRI time
courses)
• PCA and ICA of voxels × scans matrix
• Structural equation modeling to analyze
connectivity
• Pattern recognition to segment gray/white/CSF
• Meta-analysis to combine locations of
activation from different studies
Tree methods: Automatic
Interaction Detection (AID)
Morgan, J.N. and Sonquist, J.A. (1963). Problems in the
analysis of survey data, and a proposal. Journal of the
American Statistical Association, 58, 415-434.
Kass, G.V. (1980). An exploratory technique for
investigating large quantities of categorical data.
Applied Statistics, 29, 119-127.
Worsley, K.J. (1978). Significance testing in Automatic
Interaction Detection (AID). PhD Thesis, University
of Auckland.
How AID works:
• Split observations into two groups according to the
values of a predictor
• Two types of predictors:
– Monotonic: split by thresholding:
{predictor ≤ x} | {predictor > x}
– Free: split into any two subsets, e.g. if predictor
takes values {x1, …, x7}:
{x1, x5, x6} | {x2, x3, x4, x7}
• Choose the split that maximizes a test statistic for the
difference in dependent or target variable
• Repeat on two subgroups until some stopping
criterion is reached (split is not ‘significant’ or
subgroup size is too small)
SPSS example: credit risk data
Dependant or target
M
M
F
F
Predictors
M
M
F
F
M
M
M = monotonic (split by thresholding), F = free (split into any two subsets)
Brain mapping example: cortical thickness
Dependant or target
M
M
Predictor
M
M
M
Subject Node1 Node2 Node3 Node4 … Node40962 Sex
1
2
3
4
5
6
7
8
9
:
321
3.73
2.95
2.30
2.64
2.39
3.26
2.68
3.60
3.27
:
4.10
3.05
1.17
1.23
2.19
2.76
1.85
2.52
3.66
1.43
:
2.67
3.93
3.33
2.56
2.57
2.51
3.31
3.23
2.90
2.88
:
2.83
2.30
2.75
1.20
2.25
2.82
1.70
2.30
2.25
1.81
:
1.78
…
…
…
…
…
…
…
…
…
…
1.59
1.03
1.46
1.29
1.02
1.65
1.47
1.79
2.14
:
1.70
m
f
f
m
f
f
m
m
f
:
f
Misclassification matrix:
cortical thickness
Actual category
Male
Female
Predicted Male
145
18
category Female
18
140
fMRI data: 120 scans, 3 scans each of hot, rest, warm, rest, hot, rest, …
First scan of fMRI data
Highly significant effect, T=6.59
1000
hot
rest
warm
890
880
870
500
0
100
200
300
No significant effect, T=-0.74
820
hot
rest
warm
0
800
T statistic for hot - warm effect
5
0
-5
T = (hot – warm effect) / S.d.
~ t110 if no effect
0
100
0
100
200
Drift
300
810
800
790
200
Time, seconds
300
Brain mapping example: fMRI
Dependant or target
M
Frame
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
:
117
Voxel1
1.1
-0.59
1.06
1.63
2.3
1.27
1.18
0.98
1.46
0.07
0.39
0.04
-0.06
-0.48
-0.09
-0.24
-1.52
-0.07
-1.4
:
-0.01
M
Voxel2
1.66
0.23
1.57
1.79
1.96
1.36
1.33
0.9
1.25
0.7
0.68
-0.04
0.2
-0.26
-0.39
0.02
-1.11
0.1
-0.57
:
0.5
Predictor
M
M
Voxel3
1.53
0.38
1.56
0.88
1.41
0.73
1.35
0.47
0.77
1.29
1.13
-0.18
0.29
-0.19
-0.84
0.51
-1.44
-0.07
0.01
:
0.74
M
Voxel4 Voxel30786 Stimulus
0.77 ... -0.12
hot
-0.43 ... -1.73
hot
1.14 ... 0.64
hot
-0.22 ... -0.07
hot
1.33 ... 1.76
hot
0.24 ... 1.22
warm
1.3 ... 0.88
warm
0.18 ...
0.6
warm
0.73 ...
1.3
warm
1.96 ... 2.04
warm
1.81 ...
1.8
warm
0.37 ... 1.63
hot
0.49 ...
0.7
hot
-0.16 ... -0.42
hot
-0.94 ... -0.68
hot
1.2 ... 1.38
hot
-1.88 ... -1.11
hot
-0.24 ... 0.17
warm
0.3 ... 0.41
warm
:
:
:
0.83 ... 0.99
warm
Misclassification matrix:
fMRI
Predicted Hot
category Warm
Actual category
Hot
Warm
51
1
7
58
Splitting the SPM itself:
Dependant or target
?
Voxel
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
:
30786
x
1.1719
3.5156
5.8594
1.1719
3.5156
5.8594
1.1719
3.5156
5.8594
1.1719
3.5156
5.8594
1.1719
3.5156
5.8594
1.1719
3.5156
5.8594
:
.
Predictor
?
y
-10.5469
-10.5469
-10.5469
-8.2031
-8.2031
-8.2031
-5.8594
-5.8594
-5.8594
-10.5469
-10.5469
-10.5469
-8.2031
-8.2031
-8.2031
-5.8594
-5.8594
-5.8594
:
.
?
z
7.2921
7.2921
7.2921
7.2921
7.2921
7.2921
7.2921
7.2921
7.2921
14.2921
14.2921
14.2921
14.2921
14.2921
14.2921
14.2921
14.2921
14.2921
:
.
T statistic
5.4852
5.9170
5.0115
6.1082
6.4825
5.7299
6.7113
7.3540
6.5934
5.4519
6.3674
6.3184
6.2774
6.5888
6.2456
6.3583
6.4093
5.8665
:
.
How do we split on a spatial predictor?
Splits can be regarded as models with
different means for the two groups:
SPM model
SPM model
Monotonic predictor
Free predictor
Smoothed SPM model
Unsmoothed SPM model
Smooth SPM
with a filter
that matches
the model
Spatial predictor
‘Free’ predictor
So …
• Treating spatial location as a free predictor (for
the smoothed SPM) is equivalent to simply
thresholding the smoothed SPM
• We can choose the threshold to control the
false splitting rate to P < 0.05 using Bonferroni
corrections or random field theory
• If model width is unknown, we can make filter
width another parameter of the model, which
leads to scale space:
Scale space: smooth X(t) with a range of filter widths, s
= continuous wavelet transform
adds an extra dimension to the random field: X(t, s)
Scale space, no signal
S = FWHM (mm, on log scale)
34
8
6
4
2
0
-2
22.7
15.2
10.2
6.8
-60
-40
-20
0
20
One 15mm signal
40
60
34
8
6
4
2
0
-2
22.7
15.2
10.2
6.8
-60
-40
-20
0
t (mm)
20
40
60
15mm signal best detected with a ~15mm smoothing filter
Matched Filter Theorem (= Gauss-Markov Theorem):
“to best detect a signal + white noise, filter should match signal”
10mm and 23mm signals
S = FWHM (mm, on log scale)
34
8
6
4
2
0
-2
22.7
15.2
10.2
6.8
-60
-40
-20
0
20
40
Two 10mm signals 20mm apart
60
34
8
6
4
2
0
-2
22.7
15.2
10.2
6.8
-60
-40
-20
0
t (mm)
20
40
60
But if the signals are too close together they are
detected as a single signal half way between them
Scale space can even separate two signals at the same location!
8mm and 150mm signals at the same location
10
5
S = FWHM (mm, on log scale)
0
-60
170
-40
-20
0
20
40
60
113.7
20
76
50.8
15
34
10
22.7
15.2
5
10.2
6.8
-60
-40
-20
0
t (mm)
20
40
60
FWHM = 6.8mm
FWHM = 9mm
FWHM = 11mm
FWHM = 15mm
FWHM = 20mm
FWHM = 26mm
FWHM = 34mm
FWHM
FWHM
FWHM
FWHM
FWHM
FWHM
FWHM
FWHM
Functional connectivity
• Measured by the correlation between residuals
at every pair of voxels (6D data!)
Activation only
Voxel 2
++
+ +++
Correlation only
Voxel 2
Voxel 1
+
+
++
+
Voxel 1
+
• Local maxima are larger than all 12 neighbours
• P-value can be calculated using random field
theory
• Good at detecting focal connectivity, but
• PCA of residuals x voxels is better at detecting
large regions of co-correlated voxels
|Correlations| > 0.7,
P<10-10 (corrected)
First Principal
Component > threshold
False Discovery Rate (FDR)
Benjamini and Hochberg (1995), Journal of the Royal Statistical Society
Benjamini and Yekutieli (2001), Annals of Statistics
Genovese et al. (2001), NeuroImage
• FDR controls the expected proportion of false
positives amongst the discoveries, whereas
• Bonferroni / random field theory controls the
probability of any false positives
• No correction controls the proportion of false
positives in the volume
Signal + Gaussian
white noise
Signal
Noise
P < 0.05 (uncorrected), T > 1.64
5% of volume is false +
4
4
2
2
0
-2
-4
FDR < 0.05, T > 2.82
5% of discoveries is false +
True +
False +
0
-2
-4
P < 0.05 (corrected), T > 4.22
5% probability of any false +
4
4
2
2
0
0
-2
-2
-4
-4
Comparison of thresholds
• FDR depends on the ordered P-values:
P1 < P2 < … < Pn. To control the FDR at a = 0.05, find
K = max {i : Pi < (i/n) a}, threshold the P-values at PK
Proportion of true + 1
0.1 0.01 0.001 0.0001
Threshold T 1.64 2.56 3.28 3.88 4.41
• Bonferroni thresholds the P-values at a/n:
Number of voxels 1
10 100 1000 10000
Threshold T 1.64 2.58 3.29 3.89 4.42
• Random field theory: resels = volume / FHHM3:
Number of resels
0
1
10 100 1000
Threshold T 1.64 2.82 3.46 4.09 4.65
P < 0.05 (uncorrected), T > 1.64
5% of volume is false +
FDR < 0.05, T > 2.66
5% of discoveries is false +
P < 0.05 (corrected), T > 4.90
5% probability of any false +