Multi-level block permutation
Anderson M. Winkler1, Diego Vidaurre2, Matthew A. Webster1, Mark W. Woolrich2, omas E. Nichols1,3, Stephen M. Smith1
1. Oxford Centre for Functional MRI of the Brain (FMRIB), University of Oxford, UK. 2. Oxford Centre for Human Brain Activity
(OHBA), University of Oxford, UK. 3. Department of Statistics & Warwick Manufacturing Group, University of Warwick, Coventry, UK.
1 Introduction
3 Results
If each observation is a subject, and if the dependency reflects kinship, then
this dependence is equivalent to the heritability, i.e., the fraction of the
observed variance that is aributable to kinship. e simulations shown
here used varied degrees of heritability (h2 = 0, 0.4 and 0.8), as well as the
introduction of signal.
Under weak and reasonable assumptions, mainly that the data are
exchangeable under the null hypothesis, permutation tests can provide
exact control of false positives and allow the use of various non-standard
statistics. ere are, however, various common examples where simple
exchangeability can be violated, including paired tests, tests that involve
repeated measurements, when subjects are relatives (members of pedigrees),
and any dataset with known dependence among subjects or observations. In
these cases, certain permutations would create realisations that would be
incompatible with the original data even under the null hypothesis, and
thus, that cannot be used to construct the reference distribution.
Dataset A
Shuffling freely
Figure 4 shows the results. For both datasets (A and B), shuffling that
ignores the structure between observations caused the error rate not to be
controlled as the dependence became stronger. Shuffling respecting the treelike structure between the observations controlled the error rate at the
nominal level (here, 0.05).
To allow permutation inference in such cases, we propose to test the null
hypothesis using only a subset of all otherwise possible permutations, i.e.,
using only the rearrangements of the data that respect the
exchangeability, thus retaining the original joint distribution unaltered.
Instead of defining exchangeability at the level of each datum, we assert
exchangeability for blocks of data, either within block (observations are
shuffled inside each block only) or between blocks (without permuting the
observations inside, but the blocks as whole). Importantly, we also allow
nested blocks to be defined, in a hierarchical, multi-level fashion. Our
proposal does not require modelling explicitly the degree of dependence
between observations; this dependence is implicitly accounted for by the
permutation scheme.
Without signal
h2 = 0
300
1
1
2
1
1
2
1
1
2
0.4
0.6
0.8
h2 = 0.4
300
0.2
0.4
0.6
0.8
Power gain (%)
0.6
0.8
1
2500
2000
1500
300
1000
500
100
0.2
0.4
0.6
0.8
0
0
1
0.2
0.371 (0.358-0.385)
0
0
1
0.2
0.8
1
2500
2000
1500
400
300
1000
0.6
3000
500
200
0.4
0.206 (0.195-0.218)
600
1500
300
0.8
700
2000
400
0.6
800
2500
500
0.4
0.045 (0.039-0.051)
3000
600
h2 = 0.8
0.4
3000
400
1000
200
500
0.2
0.4
0.6
0.8
0
0
1
500
100
0.2
0.092 (0.084-0.100)
0.4
0.6
0.8
0
0
1
0.2
0.412 (0.398-0.425)
0.4
0.6
0.8
0
0
1
0.2
0.043 (0.038-0.049)
0.4
0.6
0.8
1
0.185 (0.174-0.196)
Dataset B
Shuffling freely
Without signal
h2 = 0
300
500
0.2
0.4
0.6
0.8
0
0
1
0.2
h2 = 0.4
0.8
0
0
1
0.6
0.8
0
0
1
0.2
0.8
1
2500
2000
500
1500
400
300
1000
0.6
3000
600
1500
0.4
0.102 (0.094-0.111)
700
200
1000
200
500
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0
0
1
0.365 (0.351-0.378)
0.6
0.8
0
0
1
0.2
1
2000
1500
400
300
1000
0.8
2500
500
200
0.6
3000
600
1500
0.4
0.093 (0.086-0.102)
700
2000
300
0.4
800
2500
400
0.2
0.044 (0.039-0.050)
3000
500
500
100
0
0
1
600
1000
200
500
100
0
0
0.4
800
2000
300
0.2
0.045 (0.040-0.051)
2500
700
h2 = 0.8
0.6
3000
800
Average Hamming distance
0.4
0.355 (0.342-0.368)
400
500
100
0.057 (0.051-0.064)
1.05
1000
200
100
−40
1500
400
1000
500
5
2000
500
200
0
0
2500
600
1500
300
3000
700
2000
400
With signal
800
2500
700
6
Without signal
3000
800
−30
Shuffling respecting structure
With signal
0.048 (0.043-0.054)
1
0.2
0.208 (0.197-0.220)
500
0
0
1
700
8
0.95
0
0
1
600
500
800
2
0.9
0.8
700
0.058 (0.052-0.065)
−20
0.85
0.6
200
100
−10
0.8
0.4
800
1000
600
0.75
0.2
0.042 (0.037-0.048)
200
1
0.7
0
0
1
1500
400
0
0
0.65
0.8
2000
500
100
−50
0.6
0.6
2500
600
10
7
0.4
3000
700
R2 = 0.898924
3
0.2
0.359 (0.345-0.372)
500
4
500
100
0
0
1
800
20
Figure 1: The dependence structure between observations (here, simulated sib-pairs)
can be represented as a table, in which each column indicates a level, or as a tree.
-1
-1
-1
-2
-2
-2
-3
-3
-3
0.2
600
e structure in each of these datasets and the possibilities for
rearrangements can be represented graphically as depencence trees, as
shown in Figure 1. e complexity of the trees contrast with the case
where shuffling is allowed freely between all observations, as shown in
Figure 2.
Dataset A
500
700
0
1000
200
800
Dataset A: ree sets of three siblings, each comprising a pair of
monozygotic twins and a non-twin.
Dataset B: ree sets of three siblings, each with a different structure: one
with a pair of monozygotic twins and a non-twin, one with a pair of
dizygotic twins and a non-twin, and one with three non-twin siblings.
300
0.053 (0.047-0.059)
Figure 5: Relationship between power gain (compared to free shuffling
when there is no heritability) and the amount of perturbation (average
Hamming distance, normalised to unity using the theoretical upper
limit as the reference) of the observations at each shuffling. The blots 1-6
are the strucutres shown in Figure 3; the blots 7-8 in Figure 6.
1500
400
1000
100
0
0
2000
500
200
0
0
2500
600
1500
400
3000
700
2000
500
With signal
800
2500
600
100
To evaluate our method, we simulated two simple datasets that reproduce
the structured dependence in the data of the Human Connectome Project
(HCP). ese two sets consisted of:
Without signal
3000
700
Although power cannot be considered when the error rate is not controlled,
it is clear that, even in the absence of true dependence, shuffling within
block is less powerful than shuffling freely. is reduction in power stems
from the reduced amount of perturbations caused on the data by the
shuffling process. is "amount of perturbations" can be measured by the
average Hamming distance, i.e., the average number of observations that
change their position at each permutation. When the variation in power is
contrasted with the average Hamming distance, as shown in Figure 5, it
becomes clear that the less perturbation, less power.
Shuffling respecting structure
With signal
800
0
0
2 Method
1
1
1
1
1
1
1
1
1
Figure 4: Error type I (in blue) and power (red or green) when shuffling respecting
or not the data structure. Red bars are for power when the error rate is not controlled
(flagged by red bars), thus when the test is not valid. The proportion of discoveries is
in bold, and the 95% confidence intervals between parenthesis.
0.2
0.4
0.6
0.8
0
0
1
500
100
0.2
0.077 (0.070-0.084)
0.4
0.6
0.8
0
0
1
0.387 (0.374-0.401)
0.2
0.4
0.6
0.8
0
0
1
0.2
0.041 (0.036-0.047)
0.4
0.6
0.8
1
0.098 (0.090-0.107)
Dataset B
-1
-1
-1
-1
-1
-1
-1
-1
-1
1
2
3
4
5
6
7
8
9
-1
-1
-1
-2
-2
-2
3
3
3
1
1
2
1
1
2
1
2
3
4 Data structure in the Human Connectome Project
1
2
3
4
5
6
7
8
9
e HCP recruits subjects together with their siblings, with emphasis on twins (monozygotic and dizygotic). As before, this structure can be represented as a
tree that indicates which pieces of data can be shuffled for inference, and the permutation methods described this far can therefore be applied. Depending on
whether there is interest in considering or not common effects in DZ twins, these can be treated as a category on their own Figure 6 (le), or be allowed to
be exchanged with ordinary siblings (right).
Figure 6: Dependence structure of the HCP data. On the le, DZ twins are treated as a category on its own; on the right, DZ twins are treated as ordinary siblings (non twins).
27
Observations
(subjects)
9
80
29
151
99
173
181
3
144
168
144
80
26
109
41
38
1
11
38
181
196
117
109
151
29
39
162
216
172
22
63
173
133
9
39
196
56
231
27
76
133
8
42
48
46
56
99
171
168
7
186
18
52
26
23
42
52
165
170
231
72
3
13
4
216
170
79
186
5
132
154
132
2
55
214
23
64
169
8
161
107
161
154
150
113
76
32
97
72
207
119
48
97
24
Branches that
begin at red dots are
not exchangeable
9
218
150
218
18
224
93
107
116
79
201
77
93
113
155
68
138
214
64
32
147
116
207
8
169
119
230
157
201
197
74
24
13
84
224
68
61
156
147
7
157
158
187
155
230
152
12
188
74
94
176
12
197
6
61
45
165
159
175
54
115
138
3
77
140
200
55
6
3
156
101
152
140
20
208
219
1
1
125
163
45
158
205
134
188
217
167
43
5
130
209
200
6
143
121
130
2
175
198
226
43
15
121
187
19
5
208
31
148
191
85
141
84
5
192
Branches that
begin at blue dots
are exchangeable
1
31
164
217
171
135
167
131
193
159
226
22
128
8
19
148
191
117
115
15
141
199
198
192
143
135
124
193
162
219
94
202
163
128
125
205
4
46
199
123
124
101
213
202
114
123
172
6
222
213
111
114
54
Figure 2: If the data is freely exchangeable, the structure is much simpler.
228
41
222
209
11
176
108
63
111
9
10
182
110
220
232
228
98
108
164
Free shuffling
195
16
10
20
134
145
96
178
98
131
3
146
17
16
85
92
4
1
1
1
1
1
1
1
1
1
1
2
3
4
5
6
7
8
9
96
28
127
17
83
92
30
2
82
28
35
232
2
83
69
166
142
40
118
30
60
67
47
65
180
110
82
182
120
189
4
35
223
5
212
69
112
136
40
229
220
67
53
166
47
65
184
112
227
183
212
211
60
136
203
2
204
229
223
211
7
1
185
185
105
195
203
189
180
178
225
204
105
57
81
174
126
145
126
190
62
102
106
4
81
190
104
146
6
225
184
58
59
104
62
90
21
227
14
127
221
137
89
58
59
90
142
9
14
137
221
25
7
120
153
57
118
7
53
153
88
102
8
103
183
75
91
174
103
106
88
8
51
21
194
89
In addition to the two family configurations above, used to examine error
type I rates and power, we also constructed various other cases of
dependency to assess in more detail the influence of the permutation
scheme over power. ese are shown in Figure 3.
25
194
206
95
7
100
33
206
95
149
91
75
86
86
160
51
71
78
34
160
78
70
100
129
33
122
129
215
215
149
36
139
179
139
44
71
177
73
73
210
34
177
Figure 3: The various dependence structures used to examine power.
37
49
66
70
87
50
87
122
210
66
44
36
179
49
50
1
2
3
4
3
6
7
2
37
6
7
5
5
5
8
4
3
8
4
3
9
1
2
9
1
2
5 Conclusion
1
6
9
12
7
12
10
6 References
10
8
5
14
3
10
6
11
5
4
11
4
11
8
7
6
8
5
1
4
2
3
6
17
8
5
4
1
13
2
7
9
16
18
9
12
7
12
15
6
10
11
Multi-level block permutation effectively controls the rate of type I errors,
even in the presence of strong dependence between observations, and can
be used as a general inference tool when the dependence structure can be
organised in blocks.
3
13
14
1
21
2
32
16
20
9
19
15
15
31
14
29
25
10
24
11
28
23
30
27
16
18
13
26
12
22
17
ere is an unavoidable loss of power due to insufficient shuffling, although
in large datasets, with relatively complex dependence structure as the HCP,
this loss might not be substantial.
Nichols TE, Holmes AP. Nonparametric permutation tests for functional
neuroimaging: a primer with examples. Hum Brain Mapp. 2002 Jan;15(1):1-25.
Van Essen DC, Smith SM, Barch DM, Behrens TE, Yacoub E, Ugurbil K, WU-Minn HCP
Consortium. e WU-Minn Human Connectome Project: an overview.
Neuroimage. 2013 Oct 15;80:62-79.
Winkler AM, Ridgway GR, Webster MA, Smith SM, Nichols TE. Permutation
inference for the general linear model. Neuroimage. 2014 May 15;92:381-97