Sept. 27, 2007 LEC #1
I.
ECON 140A/240A-1
Exploratory Data Analysis
L. Phillips
I. Introduction
At the beginning of the course we will study three branches of statistics: (1) data
analysis, (2) probability, and (3) statistical inference.
Data analysis is the gathering, display and summary of data. We will use visual
devices and quantitative measures to accomplish these tasks.
Probability has its origins in gambling and the laws of chance. This topic is
interesting in its own right but we will also use probability as a means to better
understand the binomial distribution, the central limit theorem, and the relationship
between the binomial distribution and the normal distribution.
II. Data Description
One use of statistics is to describe data with summary measures. Two notions are
central tendency and dispersion.
There are several measures of central tendency. An intuitive and relative easy
measure to use is the mode, i.e. the data value that is observed most frequently. Of course
one issue is what if the data has two or three modes and has multiple peaks.
Another measure of central tendency is the median. The data can be sorted and
ordered from the highest value to the lowest, and the data point in the middle is the
median, with one half of the data values above and one half of the data values below.
Another measure of central tendency requiring some arithmetic is the sample
mean of the data. Add up all the data values and divide by the number of observations or
data points.
III. Exploratory Data Analysis
Sept. 27, 2007 LEC #1
ECON 140A/240A-2
Exploratory Data Analysis
L. Phillips
John Tukey developed exploratory data analysis to visually describe the
characteristics of data. Two visual tools useful for this purpose are the stem and leaf
diagram and the box and whiskers plot.
An example of the methodology of the stem and leaf plot is its application to
weight data from males and females at Penn State, taken from Larry Gonick & Woolcott
Smith, The Cartoon Guide to Statistics(1993).
Males: 140 145 160 190 155 165 150 190 195 138 160 155 153 145 170 175 175 170 180
135 170 157 130 185 190 155 170 155 215 150 145 155 155 150 155 150 180 160 135
160 130 155 150 148 155 150 140 180 190 145 150 164 140 142 136 123 155
Females: 140 120 130 138 121 125 116 145 150 112 125 130 120 130 131 120 118 125
135 125 118 122 115 102 115 150 110 116 108 95 125 133 110 150 108
For this illustration, the data is pooled without regard to gender. The first step is to
determine the range of the data, the minimum weight and the maximum weight, 95 and
215, respectively. The second step is to construct the stem, counting by tens from 9 for
90, 10 for 100, etc. out to 21 for 210.
----------------------------------------------------------------------------------------------------9
10
11
12
13
14
15
Sept. 27, 2007 LEC #1
ECON 140A/240A-3
Exploratory Data Analysis
L. Phillips
16
17
18
19
20
21
Figure 1 : Stem of the Stem and Leaf Diagram
----------------------------------------------------------------------------------------------------------The third step is to construct the leaves: use the second digit of 95, the lowest weight,
which is placed after 9 on the stem. There are three weights between 100 and 110: 102,
108, and 108 so the digits following 10 on the stem are 2, 8, 8. This is a leaf attached to
the stem at 10. Continuing in this fashion:
-----------------------------------------------------------------------------------------------------------9: 5
10: 2 8 8
11: 6 2 8 8 5 5 0 6 0
12: 3 0 1 5 5 0 0 5 5 2 5
13: 8 5 0 5 0 6 0 8 0 0 1 5 3
14: 0 5 5 5 8 0 5 0 2 0 5
15: 5 0 5 3 7 5 5 0 5 5 0 5 0 5 0 5 0 0 5 0 0 0
16: 0 5 0 0 0 4
17: 0 5 5 0 0 0
18: 0 5 0 0
Sept. 27, 2007 LEC #1
ECON 140A/240A-4
Exploratory Data Analysis
L. Phillips
19: 0 0 5 0 0
20:
21: 5
Figure 2: Preliminary Leaves in the Stem and Leaf Diagram
--------------------------------------------------------------------------------------------------------The last step is to order the digits composing the leaves. This provides a visual
description of the data including the minimum, the maximum, the modes and the median.
---------------------------------------------------------------------------------------------------------9: 5
10: 2 8 8
11: 0 0 2 5 5 6 6 8 8
12: 0 0 0 1 2 3 5 5 5 5 5
13: 0 0 0 0 0 1 3 5 5 5 6 8 8
14: 0 0 0 0 2 5 5 5 5 5 8
15: 0 0 0 0 0 0 0 0 0 0 3 5 5 5 5 5 5 5 5 5 5 7
16: 0 0 0 0 4 5
17: 0 0 0 0 5 5
18: 0 0 0 5
19: 0 0 0 0 5
20:
21: 5
Figure 3: Stem and Leaf Diagram
Sept. 27, 2007 LEC #1
ECON 140A/240A-5
Exploratory Data Analysis
L. Phillips
Of course this back of the envelope technology could be combined with using a computer
to sort or order the data.
In all there are 92 observations or data points. So the median would lie between
the 46th and 47th observation, i.e. between 145 and 145 so the median is 145. Note the
data is bimodal with ten 150’s and ten 155’s. The students have a reporting bias tending
to round off to zeros and fives.
IV. Dispersion
One measure of dispersion is the interquartile range, IQR. Sort the data and put the
points into four groups with equal numbers of observations. There will be two groups
above the median and two groups below the median. If the median is a data point, add it
to both the upper group and the lower group. In the case of the weight data, we had an
even number of observations, and the median fell between two observations, the 46th and
the 47th, which were both equal to 145. Next, find the median for the two high groups,
i.e. the third quartile with 25 percent of the observations above it. Also find the median
for the two lowest groups, i.e. the first quartile with 25 percent of the observations below
it. The difference between the median for the highs and the median for the lows is the
interquartile range.
Having already done the work for the weight data by constructing the stem and leaf
diagram, we can use it to determine the first quartile of 125 pounds, between the 23rd
observation of 125 pounds and the 24th observation of 125 pounds. The third quartile is
between the 23rd and 24th observation from the top, i.e. between 157 pounds and 155
pounds so the third quartile is 156 pounds, and the interquartile range is 156 minus 125 or
31 pounds.
Sept. 27, 2007 LEC #1
ECON 140A/240A-6
Exploratory Data Analysis
L. Phillips
John Tukey’s box and whiskers plot displays the interquartile range as well as other
features of the data such as outliers. The left edge of the box is the first quartile and the
right edge of the box is the third quartile. The median is drawn as a vertical line dividing
the box.
---------------------------------------------------------------------------------------------------
125
145
156
Figure 4: Box of the Box and Whiskers Plot
------------------------------------------------------------------------------------------------
To illustrate the whiskers, we need to redraw the box at half scale horizontally so
that we have sufficient room. The whiskers end with points that are not outliers, and the
data points that are outliers are illustrated individually. Outliers are any data points that
are beyond 1.5 times the interquartile range, i.e. 1.5 times 31 or 46.5, from either end of
the box. So the first quartile of 125 minus 46.5 is 78.5, but this lies far below the
minimum point of 95 so the left whisker will end at 95 with no low outlying points. The
third quartile of 156 plus 46.5 is 202.5 so the right whisker ends at 195, the next point
below and there is one outlier at 215 pounds.
Sept. 27, 2007 LEC #1
95
125
ECON 140A/240A-7
Exploratory Data Analysis
145
156
L. Phillips
195
215
Figure 5: Box and Whiskers Plot
-----------------------------------------------------------------------------------------------------------Another measure of dispersion or spread in the data is the standard deviation, s.
This is the square root of the sample variance, i.e. the average of the squared distance of
each observation value from the sample mean:
[xj – ( xj /n)]2 }/n-1 = s2 .