Do The Ordinal Orders of Hierarchical
Complexity Produce Significant Gaps
Between Stages and Are the Stages Equally
Spaced?
Michael Lamport Commons
Harvard Medical School
Eva Yuja Li
Dare Institute
Presented at Piaget Society, Saturday, 9:00-10:30, June 2, 2012
SY21 Symposium Session 21, Symposium: Theory And Application
Generated By The Model Of Hierarchical Complexity
Conference Room B, Sheraton Centre, Toronto, CA
Background
• Model of Hierarchical Complexity is applied to many
domains to measure the complexity of tasks
– Tasks are assigned integer numbers called Orders of
Hierarchical Complexity
• As a measurement scale, its property is investigated
in this paper
– Is the Order of Hierarchical Complexity an ordinal scale?
• Only an ordinal scale is meaningful
– Are orders of hierarchical complexity linear and equally
spaced?
• A linear and equally spaced scale would indicate that there is equal
amount of difficulty to move from one stage to the next
2
Background
• Is the Order of Hierarchical Complexity an ordinal scale?
– As shown in the Rasch Variable Map, the stages of performance of
items followed the same sequence of their orders of Hierarchical
Complexity
– In addition, ordinality might show up as gaps been the stages of
performance on those items
• Are orders of hierarchical complexity linear and equally
spaced?
– Equally spaced orders would indicate that moving from one stage to the
next is always the same difficulty
– If the performance measured by Rasch Analysis on the items are
equally spaced, then the orders of Hierarchical Complexity would also
be equally spaced
3
Method
Instrument
• This study used the laundry instrument that was based on the Inhelder and
Piaget’s (1958) pendulum task.
• 111 items ranging from Primary Order 7 to Systematic Order 11
Procedure
• Instrument was presented in a survey online
• The tasks were presented in a sequence from easy to hard
• The items were coded as correct or incorrect with missing answers being
assumed incorrect
• Data was analyzed using the Rasch Model
4
Method
• Participants:
–
–
–
–
113 participants
47 (41.6%) men and 66 (58.4%) women
Age 18 to 100 (M = 35.8, (S.D. = 16.1).
Education
•
•
•
•
•
35 high school graduates
57 Bachelor’s degree holders
8 master’s level degree holders and
13 doctoral level degree holders
M = Bachelors degree
5
Results
• Rasch Analysis yielded two scales
– Person Stage of Performance
• Stands for how well the person performs on the set of tasks
• Based solely on whether or not a given order of hierarchical Complexity is
correctly carried out
– Rasch Scaled Item Difficulty
• How difficult items were empirically
• This is the focus of this study
6
Participants MAP OF RANKS
<more>|<rare>
7
+
|
|
# T|
6
+
|
## |
|
5
.## +
.## |T
.# |
### |
4
.###### + sy
#### |
S| sy sy
## | sy sy
3
### +
## | sy sy
# | sy sy
.## |S sy sy
2
### + sy sy
######## |
### | f1 f1
### | f1 f1
1
## + f1 f1
# |
M|
######## |
0
.##### +M ab ab
## | ab ab
# | ab ab
.## | ab ab
-1
. +
|
.####### |
.# | cr cr
-2
. + cr cr
. S|S cr cr
. | cr
.######### |
-3
.#### + pr
.## | pr pr
.## | pr pr
# | pr pr
-4
# + pr
# |
| pr pr
. |T
-5
. T+
. |
|
|
-6
. +
<less>|<frequ>
Rasch Variable Map
sy
sy
sy
sy
sy
sy sy
sy sy sy sy (Systemtatic)
sy sy sy sy sy
sy
f1 f1
f1 f1
f2 f2
f2
f2
f2
f2
f2
f2 f2
f2 (formal)
ab ab
ab ab
ab
ab
ab
ab
ab
ab
ab
ab (abstract)
cr cr
cr
• Rasch Scaled Item Difficulty
was ranged from -4.56
(Primary 7) to 3.94 (Systematic
11)
• The higher an item is on
the scale, the more
difficult it is
• P-Primary 7, C - Concrete 8,
A- Abstract 9, F - Formal 10, S
- systematic 11
(concrete)
cr cr
pr pr
pr pr pr pr pr pr pr pr (primary)
• Rasch Scaled Item Difficulty
of items sequenced in the same
order as their Orders of
Hierarchical Complexity
• No item was out of order
7
• Are there “gaps” between Orders of
Hierarchical Complexity?
– Gaps are defined as “jumps” in Rasch Scaled Item
Difficulty from one order to the next
– Gaps may indicate that demands of tasks between
adjacent orders have significant difference
– Every other order adds a level of coordination
– The order demands are supposed to be
qualitatively different
• Hypothesis: “Gaps” are significantly larger
than the difference of Rasch Scaled Item
Difficulty between items within each order
8
Model to Test for Gaps
• Let i = the observation number, which goes from 1 to 102.
• DRi = β + a7 I 7i + a8 I8i + a9 I9i + a10 I10i + a11 I11i + εi
where
– DRi = the difference of Rasch Scaled Item Difficulty between
item I and item (i-1)
– β = the average of Gaps
– an = the difference between the average of Item Break at order n
and the average of Gap β
– Ini = {1,0} {is, is not} a difference in Rasch scores for
Hierarchical order or group n
– εi is a random variable fulfilling the Gauss Markov conditions
9
Result
• DR = 0.65500 - 0.57447 I7i - 0.58864 I8i -0.60553 I9i 0.62237 I10i - 0.58397 I11i
• This equation shows that the average of Gaps was 0.655. The
average item break at each stage was smaller than the average
gap size as shown by the an being negative.
• There are 5 null hypotheses: an = 0, n = 7, 8, 9, 10, and 11
• There are 5 alternative hypotheses: an ≠ 0, n = 7, 8, 9, 10, 11.
10
Result
• Five t-tests were conducted
–
–
–
–
–
t7 (97) = -10.014, p < 2-16 ≈ 0.00000
t8 (97) = -9.667, p < 2-16 ≈ 0.00000
t9 (97) = -10.555, p < 2-16 ≈ 0.00000
t10 (97) = -10.848, p < 2-16 ≈ 0.00000
t11 (97) = -10.499, p <2-16 ≈ 0.00000
• All the null hypotheses were rejected
• Average Item Breaks are significantly smaller than the
average Gaps
• Therefore, we have shown that Gaps exist
11
Test for Linearity and Equal Spacing
• This section investigates whether the Order of Hierarchical
Complexity was a linear and equally spaced scale
• Four models were used
–
–
–
–
Simple regression model
Lack of fit test
Model on the spacing between Rasch Scaled Item Difficulty
Perturb the linear Order of Hierarchical Complexity
12
Test for Linearity
Simple linear regression
• Simple linear regression between
Rasch Scaled Item Difficulty y and
Order of Hierarchical Complexity
x
• y = a + b*x
• x = 7, 8, 9, 10, 11 - a linear scale
• r = .983, r2 = .975
• The result shows that Item Order
of Hierarchical Complexity
predicts Rasch Scaled Item
Difficulty with r of .983
13
Test for Linearity
Simple linear regression
How to know whether a linear relationship is the best option to
describe the data?
– By comparing the variance explained by the linear
regression model to variance explained by another model.
The model that explains more variance is better
• A Lack of Fit test compares the Linear Regression Model with
the Separate Means Model
• H0: Linear Regression Model explains significantly less
variance than the Separate Means Model
• H1: Linear Regression Model and the Separate Means Model
explains equal amount of variance in the data
14
Test for Linearity
Simple linear regression
• The lack of fit test shows that F(3) = 1.944, p = 0.128
• The separate means model does not explain significantly more
variance than the linear regression model
• The null hypothesis is that the spacing is unequal is not
rejected
• The result indicates that the linear relationship between the
Task Order of Hierarchical Complexity and the Rasch Scaled
Item Difficulty is not rejected by this analysis
• The linearity assumption can still be held
15
Test for Equal Spacing
• Using a t-test, this analysis tests whether there are equal
spacing between adjacent Orders of Hierarchical Complexity
• Spacing is defined as the increment from the average of
Rasch Scaled Item Difficulties of a lower order to the average
of Rasch Scaled Item Difficulty of the next higher order
– There are four spacings as there are five Orders of Hierarchical
Complexity
• A statistical Model is constructed to account for the differences
of Rasch Scaled Item Difficulty between items
16
Test for Equal Spacing
• RD = Rasch Scaled Item Difficulty = β7 + γ8 I8i + γ9 I9i +
γ10 I10i + γ11 I11i + εi
– RD = Rasch Scaled Item Difficulty;
– Ini = {1, 0} when the item {is, is not} at the Order of
Hierarchical Complexity denoted by n. n = {7, 8, 9, 10, 11};
– β7 = is the average value of the Rasch Scaled Item Difficulty for
items in order 7
– γ8 = the estimate of the difference between the average Rasch
Scaled Item Difficulty at order 8 score and average Rasch Scaled
Item Difficulty at order 7 score
17
Test for Equal Spacing
• β7 + γ8 estimates the average Rasch Scaled Item Difficulty at
order 8
• {β7 + γ9, β7 + γ10, β7 + γ11} estimates the average Rasch
Scaled Item Difficulty at order 9, 10 and 11
• H01: The spacing between order 9 and 8 is the same as the
spacing between order 8 and 7. Or γ9 - 2γ8 = 0.
• H02: The spacing between order 10 and 9 is the same as the
spacing between order 9 and 8. Or, γ10 - 2 γ9 + γ8 = 0
• H03: The spacing between order 11 and 10 is the same as the
spacing between order 10 and 9. Or, γ11 - 2γ10 + γ9 = 0
• One sample t-tests were used to test these hypotheses
18
Test for Equal Spacing
• The result shows that we cannot reject any of these null
hypotheses:
• For H01, t(97) = 0.240, p = 0.595
• For H02, t(97) = 0.0526, p = 0.479
• For H03, t(97) = 0.7949, p = 0.214
• Therefore, we cannot reject the null hypotheses that all the
spacing between the orders is the same. This result is
consistent the result of lack of fit test, which cannot reject
linearity of the Orders of Hierarchical Complexity.
19
Test for Equal Spacing
Perturbation Test
• Tests above supported that Order of Hierarchical Complexity
as linear and equally spaced scale
• However, it was due to the lack of evidence to reject null
hypotheses, which does not prove the alternative hypotheses
• This section of the paper takes an alternative route
– We add noise to the Orders of Hierarchical Complexity scale
– We test how much noise added to the scale would reject the linearity
hypothesis
– It will show the upper limit to the deviation from a linear scale
20
Perturbation Test
Procedure
1. Take the Orders of Hierarchical Complexity {7, 8, 9, 10, 11}
2. Randomly add or subtract 0.05 (randomly selected by
computer)
3. Run a linear regression of the Rasch Scaled Item Difficulty on
the newly defined order scale, obtain r of the model
4. Repeat step 2 three more times
5. Average four r’s, obtain the average r when OHC was
perturbed with noise of 0.05
6. Repeat step 1-5 with noise level 0.1, 0.15, 0.2, … 0.45
– Stop at 0.45 because noise of 0.5 may subvert the sequence of OHC,
thus violate ordinality of the scale
21
Perturbation Test
Result
• This scatter plot is the size for perturbation versus
predictability r
• It shows that as the scale deviates from the original linear scale,
predictability decreases steadily
R
0.99
0.985
0.98
0.975
0.97
0.965
0.96
0.955
0.95
0.945
0.94
0
0.1
0.2
0.3
0.4
0.5
0.6
Perturbation
22
Perturbation Test
Result
• Using the Fisher r-to-z transformation, the significance of the
the r found in the original linear regression model and the r's
found in the new models when the Order of Hierarchical
Complexity is perturbed were assessed
• It is found that perturbing the Order of Hierarchical
Complexity by more than 0.25 produces a significant
difference in the predictability of the scale
• When the noise was 0.25, the difference was significant at the
0.1 level, with z = 1.68, p = 0.093. When noise = 0.35, the
difference was significant at 0.05 level, with z = 2.74, p =
0.006.
23
Summary
• Rasch Analysis showed that the Order of Hierarchical
Complexity is an ordinal scale, where Orders predicted the
relative difficulty of items
• Simple linear regression showed that Orders of Hierarchical
Complexity predicted Rasch Scaled Item Difficulty with an r
of
• lack of fit test showed that the linearity of the scale could not
be rejected
• Testing for Equal Spacing showed that the equal spacing
assumption could not be rejected
• Perturbing the scale by 0.25 led to a significant difference in
the predictability of the scale
24
Discussion
• The existence of gaps shows that the ordinal nature of the scale
is not just an assumption
• MHC is an equally spaced ordinal scale
– It is not an interval scale because it does not have additively or any
cancellation property
• Equal Spacing indicates that going from one order to the next
produces equal difficulty between stages
– This allows one to treat orders as real numbers, and not just indication
of relative position
• It might mean the order of Hierarchical Complexity, n, is a
measure the quantity of hierarchical information
– The minimum number of order n task may be 2n
• Given that tasks at order n + 1 are defined by and coordinate 2 or more tasks at
order n
25