More Data Tools:
Merging and Summarizing
Heike Hofmann
stat 480
Homework
• This week’s is due (either paper or electronic),
next week’s is on the website
• submission on Blackboard
• Your job: ask and answer a question of the data
Outline
• Keys in data sets
• Pivot tables:
Setting up, grouping variables, changing fields
• Missing data
Merging data
FALSE for exact matches only
• vlookup (cell, range, col_id, FALSE)
For this cell, we want to get additional information
A table specified by top left cell and bottom right cell (e.g.
A1:F65)
Excel looks for the value of cell in column A, the additional
information ins in column ‘col_id’
‘cell’ is a special column. Because it identifies values in both
tables, it is a key …
Keys
• Identify a record (so must be unique)
• Like indices on a random variable
by design of experiment • Fixed
(known in advance)
• May be single or composite
• As compared to measurements/measured
variables
Relationships
• Keys identify an object
• Keys allow us to link data from different
sources
• 1-to-1, 1-to-many, many-to-many
Identify the key
RK
NAME
FIRST
LAST
POSITION
TEAM
SALARY
SEASON
YEAR
1
Kobe Bryant, SG
Kobe
Bryant
SG
Los Angeles Lakers
$27,849,149
2012 - 2013
2013
2
Dirk Nowitzki, PF
Dirk
Nowitzki
PF
Dallas Mavericks
$20,907,128
2012 - 2013
2013
3
Amar'e Stoudemire, PF
Amar'e
Stoudemire PF
New York Knicks
$19,948,799
2012 - 2013
2013
4
Joe Johnson, SG
Joe
Johnson
SG
Brooklyn Nets
$19,752,645
2012 - 2013
2013
5
Dwight Howard, C
Dwight
Howard
C
Los Angeles Lakers
$19,536,360
2012 - 2013
2013
6
Carmelo Anthony, SF
Carmelo
Anthony
SF
New York Knicks
$19,444,503
2012 - 2013
2013
7
Pau Gasol, PF
Pau
Gasol
PF
Los Angeles Lakers
$19,000,000
2012 - 2013
2013
8
Chris Paul, PG
Chris
Paul
PG
Los Angeles Clippers
$17,779,458
2012 - 2013
2013
9
Chris Bosh, C
Chris
Bosh
C
Miami Heat
$17,545,000
2012 - 2013
2013
10
LeBron James, SF
LeBron
James
SF
Miami Heat
$17,545,000
2012 - 2013
2013
11
Dwyane Wade, SG
Dwyane
Wade
SG
Miami Heat
$17,182,000
2012 - 2013
2013
12
Deron Williams, PG
Deron
Williams
PG
Brooklyn Nets
$17,177,795
2012 - 2013
2013
13
Andrew Bynum, C
Andrew
Bynum
C
Philadelphia 76ers
$16,889,000
2012 - 2013
2013
Identify the key
RK
NAME
FIRST
LAST
POSITION
TEAM
SALARY
SEASON
YEAR
1
Kobe Bryant, SG
Kobe
Bryant
SG
Los Angeles Lakers
$27,849,149
2012 - 2013
2013
2
Dirk Nowitzki, PF
Dirk
Nowitzki
PF
Dallas Mavericks
$20,907,128
2012 - 2013
2013
3
Amar'e Stoudemire, PF
Amar'e
Stoudemire PF
New York Knicks
$19,948,799
2012 - 2013
2013
4
Joe Johnson, SG
Joe
Johnson
SG
Brooklyn Nets
$19,752,645
2012 - 2013
2013
5
Dwight Howard, C
Dwight
Howard
C
Los Angeles Lakers
$19,536,360
2012 - 2013
2013
6
Carmelo Anthony, SF
Carmelo
Anthony
SF
New York Knicks
$19,444,503
2012 - 2013
2013
7
Pau Gasol, PF
Pau
Gasol
PF
Los Angeles Lakers
$19,000,000
2012 - 2013
2013
8
Chris Paul, PG
Chris
Paul
PG
Los Angeles Clippers
$17,779,458
2012 - 2013
2013
9
Chris Bosh, C
Chris
Bosh
C
Miami Heat
$17,545,000
2012 - 2013
2013
10
LeBron James, SF
LeBron
James
SF
Miami Heat
$17,545,000
2012 - 2013
2013
11
Dwyane Wade, SG
Dwyane
Wade
SG
Miami Heat
$17,182,000
2012 - 2013
2013
12
Deron Williams, PG
Deron
Williams
PG
Brooklyn Nets
$17,177,795
2012 - 2013
2013
13
Andrew Bynum, C
Andrew
Bynum
C
Philadelphia 76ers
$16,889,000
2012 - 2013
2013
Identify the key
RK
NAME
FIRST
LAST
POSITION
TEAM
SALARY
SEASON
YEAR
1
Kobe Bryant, SG
Kobe
Bryant
SG
Los Angeles Lakers
$27,849,149
2012 - 2013
2013
2
Dirk Nowitzki, PF
Dirk
Nowitzki
PF
Dallas Mavericks
$20,907,128
2012 - 2013
2013
3
Amar'e Stoudemire, PF
Amar'e
Stoudemire PF
New York Knicks
$19,948,799
2012 - 2013
2013
4
Joe Johnson, SG
Joe
Johnson
SG
Brooklyn Nets
$19,752,645
2012 - 2013
2013
5
Dwight Howard, C
Dwight
Howard
C
Los Angeles Lakers
$19,536,360
2012 - 2013
2013
6
Carmelo Anthony, SF
Carmelo
Anthony
SF
New York Knicks
$19,444,503
2012 - 2013
2013
7
Pau Gasol, PF
Pau
Gasol
PF
Los Angeles Lakers
$19,000,000
2012 - 2013
2013
8
Chris Paul, PG
Chris
Paul
PG
Los Angeles Clippers
$17,779,458
2012 - 2013
2013
9
Chris Bosh, C
Chris
Bosh
C
Miami Heat
$17,545,000
2012 - 2013
2013
10
LeBron James, SF
LeBron
James
SF
Miami Heat
$17,545,000
2012 - 2013
2013
11
Dwyane Wade, SG
Dwyane
Wade
SG
Miami Heat
$17,182,000
2012 - 2013
2013
12
Deron Williams, PG
Deron
Williams
PG
Brooklyn Nets
$17,177,795
2012 - 2013
2013
13
Andrew Bynum, C
Andrew
Bynum
C
Philadelphia 76ers
$16,889,000
2012 - 2013
2013
just in case we have players who switch teams during the season
Combining Information
from different tables
• Goal: merge salary information (nba-espn-salaries.xls) into file
with player statistics (nba-espn-hollinger.xls)
• Plan of attack:
• combine the key information (player/season) into a single
column (this has to be the first column in the spreadsheet)
• Use vlookup to get the salary information for each player/
season
• NOTE!!!! the table for the lookup has to be SORTED!!!
• where did the vlookup fail?
Players’ performance
• The file nba-espn-hollinger.xls contains statistics on
each player’s performance for the 2008 - 2013
seasons
• We want to combine the salary information and
the players’ performances
FIRST
LAST
TEAM
GP
MPG
TS%
AST
TO
USG
ORR
DRR
REBR
PER
VA
EWA
A.J.
Price
IND
44
12.9
0.454
28
10.4
17.7
2.6
9.4
6
11.54
6.2
0.2
SEASO
N
2012
A.J.
Price
IND
50
15.9
0.454
21.4
10.2
21.8
2.3
7.8
5
10.74
-0.7
0
2011
A.J.
Price
IND
56
15.4
0.53
19.2
10.7
21.3
1.5
9.7
5.6
14.06
42.1
1.4
2010
A.J.
Price
WSH
23
26.3
0.507
32.1
8.8
17.7
2.1
9
5.5
13.69
24.2
0.8
2013
Aaron
Brooks
HOU
82
35.6
0.549
20.4
10.9
24.8
2
6.6
4.2
16.04
219.6
7.3
2010
Aaron
Brooks
HOU
80
25
0.521
19.5
10.2
21.6
2
7
4.6
12.95
58.2
1.9
2009
Aaron
Brooks
HOU
51
11.9
0.535
23
11.7
21.4
2
7.8
5.2
13.14
19.4
0.6
2008
Aaron
Brooks
HOU/
59
21.8
0.489
23.8
10.1
25.1
1.8
5.2
3.5
13.2
42.2
1.4
2011
Aaron
Brooks
SAC
38
22.6
0.568
22.7
12
17.4
1.4
8.1
4.7
12.92
24.6
0.8
2013
Aaron
Gray
CHI
56
12.8
0.508
16.5
14.3
13.6
13.5
21.7
17.6
11.49
9.5
0.3
2009
Aaron
Gray
CHI
61
10
0.529
11.9
16.5
20.6
12
18.7
15.5
14.24
33.3
1.1
2008
Aaron
Gray
NO
41
13
0.566
9.9
19.8
11.9
13.4
25.8
19.6
11.07
3.7
0.1
2011
Aaron
Gray
TOR
49
16.6
0.526
10.4
19.6
12.2
12.1
28.3
20.3
11.09
6
0.2
2012
Acie
Law
ATL
55
10.2
0.49
30.7
10.3
16.3
1.3
10.9
6.1
10.68
-2.7
-0.1
2009
Combining data sets
• we combine datasets by matching values of the key
variables
• Excel only allows one variable for matching, i.e. we
need to combine the key variables into one column
(use concatenate)
RK
NAME
FIRST
LAST
POSITION
TEAM
SALARY
SEASON
YEAR
1
Kobe Bryant, SG
Kobe
Bryant
SG
Los Angeles Lakers
$27,849,149
2012 - 2013
2013
2
Dirk Nowitzki, PF
Dirk
Nowitzki
PF
Dallas Mavericks
$20,907,128
2012 - 2013
2013
3
Amar'e Stoudemire, PF
Amar'e
New York Knicks
$19,948,799
2012 - 2013
2013
INDJohnson,
44 SG
Joe
0.454
Joe
$19,536,360
2012 - 2013
FIRST
LAST
A.J.
Price
4
A.J.
Price
A.J.
Price
A.J.
Price
6
Aaron
Brooks
5
TEAM
GP
MPG
12.9
IND
50
15.9
IND
56
15.4
Dwight Howard, C
WSH
23
26.3
Carmelo
Anthony,
SF
HOU
82
35.6
HOU
80
25
Aaron
7
Brooks
Pau Gasol, PF
Aaron
Brooks
8
HOU Paul, 51
Chris
PG
Aaron
Brooks
HOU/
59
TS%
Stoudemire PF
AST
TO
28
10.4 SG 17.7
Johnson
0.454
21.4
10.2
0.53
19.2
10.7
Dwight
0.507
Carmelo
USG
Howard
C
21.8
21.3
0.549
32.1
Anthony 8.8 SF 17.7
20.4
10.9
24.8
0.521
19.5
Pau
11.9
0.535
Chris
21.8
0.489
Gasol
10.2
23
Paul
11.7 PG 21.4
10.1
25.1
23.8
PF
21.6
ORR
DRR
REBR
2.6
9.4
Brooklyn
Nets
6
2.3
7.8
5
1.5
9.7
5.6
Los Angeles Lakers
2.1 York 9Knicks 5.5
New
2
6.6
4.2
2
7
4.6
PER
SEASO
N
11.54
6.2 2012 -0.2
$19,752,645
2013 2012
2013
10.74
-0.7
0
2011
14.06
VA
EWA
42.1
1.4
13.69
24.22012 -0.8
$19,444,503
2013
219.6
7.3
2010
12.95
58.2
1.9
2009
$19,000,000
2 Angeles
7.8 Clippers
5.2
Los
13.14
19.42012 -0.6
$17,779,458
2013
5.2
3.5
2013
2013
16.04
Los Angeles Lakers
1.8
2013
2010
13.2
2012 - 2013
42.2
1.4
2013
2008
2013
2011
Now start the
exploration …
Potential routes of
investigation
• What is the relationship between
performance (which aspect of it?) and a
player’s salary?
• Do different positions get different average
salaries?
• Do salaries vary by team?
• Do we see an age related peak in players
performance/salary?
Routes of investigation
• What is the relationship between
performance (which aspect of it?) and a
player’s salary?
•Do different positions get different
average salaries?
•Do salaries vary by team?
• Do we see an age related peak in players
performance/salary?
Pivot tables
• Dynamically aggregate and summarize the
data
• Drag and drop columns
• Measured variables in the middle, id variables
in rows and columns (and pages)
• Find it in the Excel Menu:
Insert > Pivot Table
Your Turn
• Download file nba-data-all.xls from our website
• Create Pivot tables to answer the following
questions:
• how many minutes do players play per season?
does position impact this? is there a difference
over time?
• ... identify one additional question of interest that
you can answer with the help of a pivot table
Grouping
• Useful to simplify variables with many values
• This (in excel) is called grouping
• Group ages in decades, height into
something reasonable, dates into months
"
• How? Right-Click into the Pivot table row,
choose “Group” to open the dialog window
Pivot table hints
• Choose the variable with the most
categories and put it in the rows
• Put the variable with the next most in the
columns
• “data” column, goes to far left of columns
• More than two variables is tricky, play
around, remembering what comparison
you are interested in comparing
Field settings
• Change summary type (average, sum,
standard deviation)
• Formatting
• Special calculations
Missing data
• Excel is inconsistent - sometimes missing,
sometimes error
• What happens when a blank is in a
mathematical calculation?
• What happens when a blank is in a formula?