overall
approachwhich
is much
scalable
than the
S&S and
sian distribution,
makesmore
it slightly
less general
than
method presented by Fan Li [16] guarantees
global maxisume ha represents
a vector of zeros. Ba
methods.
In
addition,
the
RB
methods
presented
in
this
CB
methods.
This
clear
computational
advantage
lead
us
to
the other methods. In addition, other works illustrates that
mum solution
inthe
all ⌦
instances.
presented vector.
in [27], itThese
is possible
to relaxasthe
J
is
and
h
is
the
potential
methods
paper
have
a
polynomial
worst
case
runtime,
while
all
other
explore
additional
Regression
based
(RB)
structure
learning
there
are
better
feature
selection
methods
than
stepwise
seThis
model
allows
exact
inference
by
using
the
GGM’s
is
zero.
However,
computing
h research
requires c
sume
h
represents
a
vector
of
zeros.
Based
on
the
tribution, which makes it slightly less general
than
method
presented
by
Fan
Li
[16]
guarantees
a
global
maximethods
have
an
exponential
worst
case
runtime.
methods.
In
addition,
the
RB
methods
presented
in
this
lection
and
backwards
selection
[19].
distribution’s
information
form
(Eq.
8).
verting
which
is expensive
eventhat
withht
r methods.
In
addition,
other
works
illustrates
that
mum
solution
in
all
instances.
sian distribution,
which makes
it slightly less
general than Parameters
method
presentedvia
Fan
guarantees
a global
maxipresented
inby
[27],
itLiis[16]
possible
to⌦,relax
the
assumption
Estimating
Building
Simulation
Bayesian
Structure
Learning
paper
have
a
polynomial
worst
case
runtime,
while
all
other
While
the
feature
selection
methods
can
be
improved,
the
sian
distribution,
which
makes
it
slightly
less
general
than
method
presented
by
Fan
Li
[16]
guarantees
a
global
maxipresented
by
A.
Dobra
et. al [5].
otherselection
methods.methods
In addition,
works
illustrates
that
mum
solution
in
all
instances.
e betterthe
feature
thanother
stepwise
seThis
model
allows
exact
inference
by
using
the
GGM’s
Tcomputing
T
is
zero.
However,
h
requires
computing
⌃ by in3.3
Regression
Based
1
2
1
p(x)
/
exp(x
Jx
+
h
x)
(8)
the
other
methods.
addition,
other
works
illustrates
mum
solution
in all
instances.
have
an
worst
case
runtime.
overall
approach
is exponential
much
scalable
than
the
S&S
andthat
Richard
E.
Edwards
R.
New
,
and
Lynne
E.
Parker
there
are
better
feature
selection
methods
than
stepwise
se- , Joshua
This
model
allows
exact
inference
by
using
the
GGM’s
andmethods
backwards
selection
[19].Inmore
distribution’s
information
form
(Eq.
8).
Overall,
A.
Dobra
et. al [5]’s
meth
verting
⌦,
which
is
expensive
even
with
the
efficient
method
there
are
better
feature
selection
methods
than
stepwise
seThis
model
allows
exact
inference
by
using
the
GGM’s
1University
RB
structure
learning
methods
determine
all
conditional
CB
methods.
This
clear
computational
advantage
lead
us
to
lection
and
backwards
selection
[19].
distribution’s
information
form
(Eq.
8).
of
Tennessee,
Electrical
Engineering
and
Computer
Science
the feature selection methods can be improved, the
and
computationally
feasible
for
a
large
J
is
the
⌦
and
h
is
the
potential
vector.
These
methods
aspresented
by
A.
Dobra
et.
al
[5].
T
T
lection
and
backwards
selection
[19].
distribution’s
information
(Eq. 8).
2Oak
3.3
Based
While
the
feature
selection
methods
can
be improved,
the
/ exp(x
+ hform
x) Integration
(8)
dependencies
by
assuming
that
eachp(x)
variable
is
a Jx
functional
explore
additional
Regression
based
(RB)
structure
learning
Ridge
National
Lab,
Whole
Building
& Community
Group
approach
isRegression
much
more
scalable
than
the
S&S
and
T Based
T
variables.
However,
the
resulting
mode
sume
h
represents
a
vector
of
zeros.
on
the
research
Overall,
A.
Dobra
et.
al
[5]’s
method
is
fairly
robust
p(x)
/
exp(x
J
x
+
h
x)
(8)
While
the
feature
selection
methods
can
be
improved,
the
overall
approach
is much
more
scalable
than
the
andrandom variables. This concept isT
methods.
In
addition,
the RB
methods
presented
in S&S
this
T
result
from
of all
hods.
This
clear
computational
advantage
lead
us
toa subset
RB
structure
learning
methods
determine
all
conditional
p(x)
/ exp(x
Jxfeasible
+the
h x)
(8)
in
[27],
it is
possible
toThese
relax
assumption
that
h
J is us
theto⌦presented
and h is the
potential
vector.
methods
asdent
upon
the
variable
ordering.
Each
overall
approach
is
much
more scalableadvantage
than the S&S
and
and
computationally
for
a
large
number
of
random
CB
methods.
This
clear
computational
lead
paper have
a polynomial
case runtime,
while all
other
additional
Regression
based worst
(RB)that
structure
learning
J
is
the
⌦
and
h
is
the
potential
vector.
These
methods
asbest
presented
in
Linear
Gaussian
Bayesian
Networks,
where
dependencies
by
assuming
each
variable
is
a
functional
sume
h represents
avariables.
vector computing
of zeros.
Based
on
the
research
is zero. However,
h requires
computing
⌃
by in-isa heavily
CB methods.
ThisRegression
clear Abstract
computational
advantage
lead
us to
explore
additional
based (RB)
structure
learning
dering
may
produce
di↵erent
joint
p
However,
the
resulting
model
depenResults
J sume
is distributed,
the h⌦ represents
and h is
theaeach
potential
ass. result
In
addition,
the
RB
methods
presented
in this variable
vector
ofvector.
zeros. These
Basedmethods
on the research
methods
have
an
exponential
worst
case
runtime.
each
random
is
Gaussian
but
defrom
a
subset
of
all
random
variables.
This
concept
is
presented
in [27], it
isdent
possible
to relax
the
assumption
that hEach
explore additional
Regression
based
(RB) structure
learning
methods.
In addition,
the RB
methods
presented
in this verting
⌦,
which
is
expensive
even
with
the
efficient
method
tion
because
each
variable
is
only depend
upon
the
variable
ordering.
unique
variable
orsume
h
represents
a
vector
of
zeros.
Based
on
the
research
presented
in
[27],
it
is
possible
to
relax
the
assumption
that
h
avebest
aMany
polynomial
worst
case
runtime,
while
all
other
key have
building
design
policies
are
made
using
sophisticated
computer
pendent
random
Gaussian
distribution
iset.
a linear
presented
Linear
Gaussian
Bayesian
Networks,
where
methods.
In
addition,
the
RBcase
methods
presented
inzero.
this
paper
ain
polynomial
worst
runtime,
whilevariable’s
all
other
is
However,
computing
h
requires
computing
⌃
by
inpresented
by
A.
Dobra
al
[5].
that
proceed
it.that
This
ensure
dering
produce
di↵erent
joint
probability
distribuis zero.
However,
harequires
computing
⌃ hbyassumption
inpresented
in
[27],may
it iscomputing
possible
to relax
the assumption
3.3
Regression
Based
s have
an
exponential
worst
case
runtime.
simulations
such
as
EnergyPlus
(E+),
the
DOE
flagship
whole-building
energy
have
exponential
case runtime.
paper have
a an
polynomial
worstworst
case
runtime,
while
all
other
verting
which
is
expensive
even
with
the
efficient
method
eachmethods
random
variable
is Gaussian
distributed,
but
each⌦,decombination
of its
parents.
Therefore,
one
can
clearly
learn
Overall,
A.
Dobra
et.
al
[5]’s
method
is
fairly
robust
verting
⌦,
which
is
expensive
even
with
the
efficient
method
is
zero.
However,
computing
h
requires
computing
⌃
by
intion
because
each
variable
is
only
dependent
upon
variables
graph
structure
is
a
valid
Bayesian
Netwo
simulation
engine.
E+
and
other
sophisticated
computer
simulations
have
RB
structure
learning
methods
determine
all
conditional
methods
have
an
exponential
worst
case
runtime.
presented
by verting
A.
Dobra
et.
al
[5].
pendent
random
variable’s
Gaussian
distribution
is
a
linear
the
structure
for
a
Linear
Gaussian
Bayesian
Network
by
and
computationally
feasible
for
a
large
number
of
random
presented
by
A.
Dobra
et.
al
[5].
Regression
Based
⌦,
which
is
expensive
even
with
the
efficient
method
several
major
problems.
The
two
main
issues
are
1)
gaps
between
the
3.3
Regression
Based
that
proceed
it.
This
assumption
ensures
that
the
resulting
method
to
search
the
set
of
all
possible
o
dependencies
by
assuming
that
each
variable
is
a
functional
Overall,
A.
Dobra
et.
al
[5]’s
method
is
fairly
robust
combination
of
its
parents.
Therefore,
one
can
clearly
learn
Overall,
A.
Dobra
et.
al
[5]’s
method
is fairly
robust
performing
a
series
of
linear
regressions
with
feature
selecvariables.
However,
the
resulting
model
is
heavily
depenpresented
by
A.
Dobra
et.
al
[5].
simulation
model
and
the
actual
structure,
and
2)
limitations
of
the
modeling
ructure
learning
methods
determine
all
conditional
3.3
Regression
Based
graph
structure
is
a
valid
Bayesian
Network,
but
requires
the
RB
structure
learning
methods
determine
all
conditional
the
best
structure.
A.
Dobra
et.
al
[5]
result
from
a
subset
of
all
random
variables.
This
concept
is
and
computationally
feasible
for
a
large
number
of
random
and
computationally
feasible
for
a
large
number
of
random
the
structure
for
a
Linear
Gaussian
Bayesian
Network
by
Overall,
A.
Dobra
et.
al
[5]’s
method
is
fairly
robust
engine’s
capabilities.
Currently,
these
problems
are
addressed
by
having
an
dent
upon
the
variable
ordering.
Each
unique
variable
ortion.
ncies
by
assuming
that
each
variable
is
a
functional
dependencies
by
assuming
that
each
variable
is aconditional
functional
method
to
search
the
set
of
all
possible
orders
to
determine
RB
structure
learning
methods
determine
all
by
scoring
the
individual
variables
and
g
best
presented
in
Linear
Gaussian
Bayesian
Networks,
where
variables.
However,
the
resulting
model
is
heavily
depenvariables.
However,
the
resulting
model
is
heavily
depenmanually
calibrate
simulation
parameters
towith
real
world
data
or
using
and may
computationally
feasible
forjoint
a large
number ofdistriburandom
a
series
of
linear
regressions
feature
selecdering
produce
a
di↵erent
probability
The
RB
approach
is
extremely
scalable.
For
example,
omperforming
aengineer
subset
of
all
random
variables.
This
concept
is
result
from
a
subset
of
all
random
variables.
This
concept
is
dependencies
by methods
assuming
that
each
variable
a functional
the
best
structure.
A.
Dobra
et.
al
[5]
addresses
this
issue
dent
the
variable
ordering.
Each
unique
variable
oreach
random
variable
is Gaussian
distributed,
butis
each
de- upon
variables
in
ascending
order
based
on
th
dent
upon
the
variable
ordering.
Each
unique
variable
oralgorithmic
optimization
to
adjust
the
building
parameters.
However,
variables.
However,
the
resulting
model
is
heavily
depentionRB
because
each to
variable
islarge
only dependent upon variables
tion.in
best
presented
in
Linear
Gaussian
Bayesian
Networks,
where
sented
Linear
Gaussian
Bayesian
Networks,
where
M.
Gustafsson
et.
al
[9]
used
methods
learn
result
from
a
subset
of
all
random
variables.
This
concept
is
dering
produce
aindividual
di↵erent
joint
probability
distribudering may
di↵erent
joint
probability
distribubyamay
scoring
the
variables
and
greedily
ordering
the
pendent
random
variable’s
Gaussian
is
a
linear
some simulations
engines, like
E+, aredistribution
computationally
expensive,
whichproduce
Fan
Li
[16]
builds
on
this
approach
and
dent
upon
the
variable
ordering.
Each
unique
variable
oreach
random
variable
isextremely
Gaussian
distributed,
but
each
de- that
proceed
it.
This
assumption
ensures
that
the
resulting
ndom The
variable
isapproach
Gaussian
distributed,
but
each de(a)
Bayesian
Estimates
(b)
Random
Estimates
RB
is
scalable.
For
example,
undirected
graphical
model
structures
in
Gene
Regulatory
best
presented
in
Linear
Gaussian
Bayesian
Networks,
where
tion
because
each
variable
is
only
dependent
upon
variables
tion
each
variable
is
only
dependent
upon
variables
makes repeatedly
evaluating
the
simulationone
engine
costly.
This
workbecause
exploresdering
variables
in
ascending
order
based
on
their
assigned
score.
combination
of
its
parents.
Therefore,
can
clearly
learn
may
produce
a
di↵erent
joint
probability
distriburequirement
without
adding
additional
a
pendent
random
variable’s
Gaussian
distribution
is
a
linear
random
variable’s
Gaussian
distribution
is
a
linear
graph
structure
is
a
valid
Bayesian
Network,
but
requires
the
M.
Gustafsson
et.
al
[9]
used
RB
methods
to
learn
large
each random
variable
is Gaussian
distributed,
but
each
dethat
proceed
it.
This
assumption
ensures
that
the
resulting
Networks.
Microarray
data
generally
contains
thousands
of
addressing
this
issue
by
automatically
discovering
the
simulation’s
internal
that
proceed
it.
This
assumption
ensures
that
the
resulting
the
structure
for
a
Linear
Gaussian
Bayesian
Network
by
tion
because
each
variable
is
only
dependent
upon
variables
Figure
1. Compares
Bayesian on
GGM’s
error
againstaaWishart
[0,
1] uniform
distribution’s
Fan
Li
[16]
builds
this
approach
and
removes
the
order
combination
of
its
parents.
Therefore,
one
can
clearly
learn
[16]
applies
prior
to
the
precisi
ation
of
its
parents.
Therefore,
one
can
clearly
learn
method
to
search
the
set
of
all
possible
orders
to
determine
pendent
random
variable’s
Gaussian
distribution
is
a
linear
graph
structure
is
a
valid
Bayesian
Network,
but
requires
the
undirected
graphical
model
structures
in
Gene
Regulatory
input and
output
dependencies
from
~20
Gigabytes
of
E+
simulation
data,
graph
structure
is
a
valid
Bayesian
Network,
but
requires
the
random
variables
and
very
few
samples.
In
that
particular
error
on
estimating
FG
building
parameters.
Additionally,
it
shows
how estimates
that
proceed
it.
This
assumption
ensures
that
the
resulting
performing
a
series
of
linear
regressions
with
feature
selecthe
structure
for
a
Linear
Gaussian
Bayesian
Network
by
requirement
without
adding
additional
assumptions.
Fan
Li
Lasso
regression
to
perform
feature
selec
cture
for
a
Linear
Gaussian
Bayesian
Network
by
its ~200
parents.
Therefore,
can clearly
learn
method
to of
search
the
set et.
of all
possible
orders
to issue
determine
the
best
structure.
A.
Dobra
al
[5]
addresses
this
future combination
extensions
willofuse
Terabytes
of E+ one
simulation
data.
The
model
isgraph
align
with
test
building
parameter
values.
Networks.
Microarray
data
generally
contains
thousands
of
method
to
search
the
set
all
possible
orders
to
determine
structure
is astructure
valid
Bayesian
Network,
but
requires
the matrix, and uses
research,
the
algorithm
for
building
the
graph
is
performing
a
series
of
linear
regressions
with
feature
selecis
formulated
as
the
following
under
ADMM:
tion.
[16]
applies
a
Wishart
prior
to
the
precision
ing avalidated
series
of
linear
regressions
with
feature
selec- Network
the
best
structure.
A.
Dobra
et.
al
[5]
addresses
this
issue
prior
to
the
precision
matrix
allows
the
m
the
structure
for
a
Linear
Gaussian
Bayesian
by
by
inferring
building
parameters
for
E+
simulations
with
ground
truth
by
scoring
the
individual
variables
and
greedily
ordering
the
the
best
structure.
A.
Dobra
et.
al
[5]
addresses
this
issue
random
variables
and
very
few
samples.
In
that
particular
s are The
methodLasso
tocoefficients
search
the
set
of
all
possible
orders
to
determine
tion.
fairly
straightforward.
If
the
regression
are
nonRB
approach
is
extremely
scalable.
For
example,
regression
to
perform
feature
selection.
Applying
the
by scoring
the
individual
variables
andprior
greedily
ordering
minimizef
+ g(z)regressions
(10) accurately
building
parameters.
results
indicate
that the model
represents
performing
a Our
series
of (x)
linear
with
feature
selecBayesian
Parameter
Estimates
Bayesian
Parameter the
Estimates
the
distribution
to
the
regression
variables
in
ascending
order
based
on
their
assigned
score.
ution
by
scoring
the
individual
variables
and
ordering
the
the
best
structure.
A.
Dobra
et.
al
[5]
addresses
this Variables
issue27 to 52
The
RB
approach
is
extremely
scalable.
For
example,
research,
the
algorithm
for
building
the
graph
structure
is
Variables
1 to 26 greedily
zero,
then
there
exists
a
dependency
between
the
response
M.
Gustafsson
et.
al
[9]
used
RB
methods
to
learn
large
in
ascending
order
based
on
their
assigned
score.
parameter
means
withx some
deviation
from
the
means,
but does not support variables
RB
isconstraint
extremely
scalable.
For
example,
tion.
prior
to
the
precision
matrix
allows
the
method
to
propagate
le to approach
with
the
z
=
0,
where
g
is
an
indicator
funcFan
Li
[16]
builds
on
this
approach
and
removes
the
order
enables
the
Lasso
regression
method to
variables
in ascending
order
based
on
their
assigned
score.
M.straightforward.
Gustafsson et. alIf[9]the
used
RB methods
to learn
largenonby
scoring
the
individual
variables
and
greedily
ordering
the
fairly
regression
coefficients
are
Estimate
Estimate
1.5 Li [16]
1.5
1.5 removes the order
1.5
inferring
parameter
values
that
exist
ongto
the
distribution’s
tail.
Fan
builds
oncoefthis approach
and
undirected
graphical
model
structures
in
Gene
Regulatory
variable
and
the
predictors
with
a
non-zero
regression
tafsson
et.
al
[9]
used
RB
methods
learn
large
The
RB
approach
is
extremely
scalable.
For
example,
tion.
Using
an
indicator
function
for
allows
ADMM
to
the
prior
distribution
to
the
regression
coefficients,
which
Actual
Actual
undirected
graphical
model
structures
in
Gene
Regulatory
requirement
without
adding
additional
assumptions.
Fan
Li
Fan
Li
[16]
builds
on
this
approach
and
removes
the
order
variables
in
ascending
order
based
on
their
assigned
score.
feature
selection.
In
addition,
applying
zero,
then
there
exists
a
dependency
between
the
response
requirement
without
adding
additional
assumptions.
Fan
Li
represent
the original
convex
optimization
constraints,
and
Microarray
data
generally
contains
thousands
of
M.
Gustafsson
et.
al
[9]
used
RB
methods
tothousands
learn
largeof directly extracts
ed Networks.
graphical
model
structures
in
Gene
Regulatory
ficient.
While
therequirement
method
an
undirected
1
1
1method to perform Bayesian
1
enables
the
Lasso
regression
Networks.
Microarray
data
generally
contains
Fan
Li
[16]
builds
on
this
approach
and
removes
the
order
without
adding
additional
assumptions.
Fan
Li
[16]
applies
a
Wishart
prior
to
the
precision
matrix,
and
uses
allows
Fan
Li
to
prove
all
possible
resul
Problem
Formulation
the
x and
z = 0 constraint
guarantees
that
the xathat
minimizes
[16]
applies
a
Wishart
prior
to
the
precision
matrix,
and
uses
variable
the
predictors
with
non-zero
regression
coefundirected
graphical
model
structures
in
Gene
Regulatory
ks. Microarray
data
generally
contains
thousands
ofdependency
random
variables
and
very
few
samples.
In
that
particular
conditional
graph,
it
may
not
be
possible
to
exrandom
variables
and
very
few
samples.
In
that
particular
feature
selection.
Infeature
addition,
applying
the
prior
to
⌦
without
adding
additional
Fan
Li
f (x) obeys the original constraints.
[16] applies
a requirement
Wishart
prior
to
the
precision
matrix,
and
0.5
0.5 assumptions.
0.5 uses
0.5 lets
Lasso
regression
to
perform
feature
selection.
Applying
the
has
Lasso
regression
to
perform
selection.
Applying
the
works
encode
the
same
GGM
joint
pro
Networks.
Microarray
data
generally
contains
thousands
of
ficient.
While
the
method
directly
extracts
an
undirected
variables
and
very
few
samples.
In
that
particular
the
algorithm
forsolution
building
the
graph
structure
isdistribution
research,
the
algorithm
for building
the
graph
structure
is prior
While
[3] used
this
general
format
to solve
o re-research,
Learning
tract
an many
overall
joint
from
the
resulting
graph.
[16]
applies
a Wishart
prior
toprove
the
precision
matrix,
and
uses
Lasso
regression
to
perform
feature
selection.
Applying
the
allows
Fan
Li
to
all
possible
resulting
Bayesian
Netto
the
precision
matrix
allows
the
method
to
propagate
prior
to
the
precision
matrix
allows
the
method
to
propagate
of
variable
order.
This
means
random
variables
and
very
few
samples.
Inisbe
that
particular
0
5
10
15
20
25
26
31
36
41
46
51 that th
di↵erent
convex
optimization
problems,
we
are
onlynot
focus,can
the
algorithm
for
building
the
graph
structure
conditional
dependency
graph,
it
may
possible
to
exfairly
straightforward.
If
the
regression
coefficients
are
nonfairly
straightforward.
If
the
regression
coefficients
are
nonLasso
regression
to perform
feature
selection.
Applying
the
prior
to
the
precision
matrix
allows
the
method
to
propagate
Variables
Variables
ollowing under
ADMM:
A.
Dobra
et.
al
[5]
focuses
on
recovering
the
factorized
the
prior
distribution
to
the
regression
coefficients,
which
works
encode
the
same
GGM
joint
probability,
regardless
E+
Building
Specification
the
prior
distribution
to
the
regression
coefficients,
which
ing
on
the
version
used
to
solve
Lasso
regression.
The
disresearch,
the
algorithm
for
building
the
graph
structure
is
SecMAP
estimate,
which
has
very
appeali
raightforward.
If the
regression
are
nonthen
there
exists
a coefficients
dependency
between
the the
response
Estimate
the
tract
an
overall
joint
distribution
from
the
resulting
graph.
zero,zero,
then
there
exists
a
dependency
between
the
response
Parameters
X
prior
to
the
precision
matrix
allows
the
method
to
propagate
prior[5]
distribution
to
the
regression
coefficients,
which
enables
the
Lasso
regression
method
to
perform
Bayesian
tributed
optimization
steps
for solving
large
scale linear
Lasso
graph’s
joint
distribution.
assumes
that
the
overall
joint
(a)
(b)
of
variable
order.
This
means
that
the
computed
⌦
is
a
enables
the
Lasso
regression
method
to
perform
Bayesian
inimizef
(x)
+
g(z)
(10)
joint
probability
fairly
straightforward.
If
the
regression
coefficients
are
nonen there
exists
a
dependency
between
response
variable
and
the
predictors
with
a
non-zero
regression
coef1 the
as
avoiding
overfitting.
In
addition,
th
A.regression
Dobra
et.predictors
alare[5]
focuses
on
recovering
the
factorized
and problems
the
with
a non-zero
regression
coefthe
prior
distribution
to
the
regression
coefficients,
which
presented
below
.
feature
selection.
In
addition,
applying
the
prior
to
⌦
lets
enables
the
Lasso
regression
method
to
perform
Bayesian
usingvariable
distribution:
distribution
is
sparse
and
represented
by
a
Gaussian
with
feature
selection.
In
addition,
applying
the
prior
to
⌦
lets
MAP
estimate,
which
has
very
appealing
properties
such
zero,
then
there
exists
a
dependency
between
the
response
ficient.
While
the
method
directly
extracts
an
undirected
and
the
predictors
with
a
non-zero
regression
coefz
=
0,
where
g
is
an
indicator
funcObserved
Output
Data
/
RealBayesian
Parameter
Estimates
Bayesian Parameter
Estimates
point
way
to
transform
the
Lasso
regression
f
enables
the
Lasso
regression
method
toresulting
perform
Bayesian
ficient.variable
While
the
method
directly
extracts
an
undirected
allows
Fan
Li
to
prove
all
possible
Bayesian
Netgraph’s
joint
distribution.
[5]
assumes
that
the
overall
joint
P(X,Y
)
feature
selection.
In
addition,
applying
the
prior
to
⌦
lets
1
⇢
2
k
k
2
k+1
Variables
53 topossible
78
Variables
79
to 104 introduces a
world
Sensor
data,
Weather
and
the
predictors
with
a
non-zero
regression
coefallows
Fan
Li
to
prove
all
resulting
Bayesian
Netator
function
for
g
allows
ADMM
to
conditional
dependency
graph,
it
may
not
be
possible
to
exN
(0,
⌃).
This
type
of
Markov
Network
is
called
a
Gaussian
as
avoiding
overfitting.
In
addition,
the
[17]
||A
x
b
||
+
||x
z
+
u
||
)
(11)
x
=
argmin
(
i
i
i
i
While
the
an
undirected
tra2
i 2
i method directly extracts
feature
selection.
In
addition,
applying
the
prior
to
⌦
lets
works
encode
the
same
GGM
joint
probability,
regardless
Data,
Operation
Schedule
Y
2
2
SVM
regression
problem,
where
the
solut
conditional
dependency
graph,
it
may
not
be
possible
to
exx
allows
Fan
Li
to
prove
all
possible
resulting
Bayesian
Netdistribution
is
sparse
and
represented
by
a
Gaussian
with
convex
optimization
constraints,
and
Estimate probability,
Estimate 1.5
ficient.
While joint
the
method
directly
extracts
an undirected
Im- dependency
tract
an overall
distribution
from
theexresulting
graph. and
works
encode
the
same
GGM
joint
regardless
1.5
1.5
1.5
way
to
transform
the
Lasso
regression
formulation
into
an
Graphical
Model
(GGM),
it
is
assumed
that
the
joint
nal
graph,
it
may
not
be
possible
to
allows
Fan
Li
to
prove
all
possible
resulting
Bayesian
Netk+1
k+1
k
of
variable
order.
This
means
that
the
computed
⌦
is
a
Actual
Actual
nt
guarantees
thatoverall
the
x that
minimizes
works
encode the same GGM joint probability,gression
regardless
tract
an
joint
distribution
from
thenot
resulting
graph.
problem
is
also
the
solution
to
t
z
=
S
(x̄
+al
ūof)[5]
(12)
abilN
(0,
⌃).
This
type
Markov
Network
is
called
a
Gaussian
A.
Dobra
et.
focuses
on
recovering
the
factorized
conditional
dependency
graph,
it
may
be
possible
to
exof
variable
order.
This
means
that
the
computed
⌦
is
a
joint
distribution
from
the
resulting
graph.
SVM
regression
problem,
where
the
solution
to
the
SVM
reprobability
is
represented
by
the
following:
MAP
estimate,
which
has
very
appealing
properties
such
aloverall
constraints.
works
encode
the
same
GGM
joint
probability,
regardless
1
1
1
1
has A.graph’s
ofgraph.
variable
order. This means that the computed
⌦ is This
a
Dobra
et.
al
[5]
focuses
on
recovering
the
factorized
k+1
k overall
k+1
k+1
problem.
transformation
allows
th
joint
distribution.
[5]
assumes
that
the
overall
joint
Graphical
Model
(GGM),
and
it
is
assumed
that
the
joint
tract
an
joint
distribution
from
the
resulting
Inference
u
=
u
+
x
z
(13)
general
solution
format
to
solve
many
obra
on recovering the factorized
MAP
estimate,
which
has
very
appealing
properties
such
i
i
as
avoiding
overfitting.
In
addition,
the
[17]
introduces
a
dif- et. ali [5] focuses
of
variable
order.
This
means
that
the
computed
⌦
is
a
gression
problem
is
also
the
solution
to
the
Lasso
regression
MAP
estimate,
which
has
very
appealing
properties
such
assumes
that
all
observations
are
independent
and
identinot
feasible
to
specify
the
single
best
graph
structure
in
ad0.5
0.5
0.5
0.5 rando
graph’s
joint
distribution.
[5]
assumes
that
the
overall
joint
distribution
is
sparse
and
represented
by
a
Gaussian
with
Estimated
Optimize
over
the
joint
A.
Dobra
et.
al
[5]
focuses
on
recovering
the
factorized
mization
problems,
we
are
only
focus1
1
nonlinear
dependencies
among
the
probability
is
represented
by
the
following:
T overfitting.
rob- distribution.
New
Observation
joint
[5]
assumes
that
the
overall
joint
way
to
transform
the
Lasso
regression
formulation
into
an
as
avoiding
In
addition,
introduces
a
MAP
estimate,
which
has
very
appealing
properties
such
cally
distributed.
Eq.
1
may
be
rewritten
as:
vance.
Therefore,
algorithmic
techniques
must
be used
to the [17]
The
individual
local
subproblems
are
solved
using
ridge
reproblem.
This
transformation
allows
the
method
to detect
Building
probability
distribution
:
exp(
(x
µ)
⌦(x
µ))
(6)
as
overfitting.
In
addition,
the
[17]
introduces
a
ed todistribution
solve
regression.
The
NLasso
(0, ⌃).
This
type
of Markov
Network
is
called
a
n Gaussian
1avoiding
Y
, sparse
Y , …,
Y disis
and
represented
by
a
Gaussian
with
graph’s
joint
distribution.
[5]
assumes
that
the
overall
joint
tting
find the2 best
graph
structure
matches
theaddition,
true joint
N Gaussian
ever,
exploring
nonlinear
dependencies
SVM
regression
problem,
where
solution
to
the
Parameters
X
gression,
and
the
global
z
values
are
computed
by
evaluating
52
57 that
62
67 In
73
78 the
78 introduces
83 into
88SVM
93 re- 98
103
tion
is
sparse
and
represented
by
a
with
as
avoiding
overfitting.
the
[17]
a
2
2
way
to
transform
the
Lasso
regression
formulation
an
⇧
P
(X,
Y
)
(2)
i
(2⇡)
|⌃|
i=1
steps
for
solving
large
scale
linear
Lasso
1
1
nonlinear
dependencies
among
the
random
variables.
Howway
to
transform
the
Lasso
regression
formulation
into
an
ationN (0,
Graphical
Model
(GGM),
and
it
is
assumed
that
the
joint
probability
distribution
without
overfitting
the
observed
trainT
Variables
Variables
distribution
is
sparse
and
represented
by
a
Gaussian
with
⌃).
This
type
of
Markov
Network
is
called
a
Gaussian
1
a soft
thresholding
function
S. This
function
is
defined
as
gression
problem
is Lasso
also
the
solution
to to
the
Lasso
regression
exp(
(x
µ)
⌦(x
µ))
(6)
.teraThis
type
of
Markov
Network
is
called
a
Gaussian
future
work,
because
naively
implement
are
presented
below
.
way
to
transform
the
regression
formulation
into
an
ing
samples.
There
are
three
categories
of
algorithms
that
which
is
the
joint
probability
distribution.
Conversely,
the
SVM
regression
problem,
where
the
solution
the
SVM
ren
1
probability
is
represented
by
the
following:
SVM
regression
problem,
where
the
solution
to
the
SVM
reever,
exploring
nonlinear
dependencies
via
this
method
is
follows:
(c)
(d)
N
(0,
⌃).
This
type
of
Markov
Network
is
called
a
Gaussian
2
2
2
Graphical
Model
(GGM),
and
it
is
assumed
that
the
joint
(2⇡)
|⌃|
problem.
This
transformation
allows
the
method
to
detect
address
this
problem.
The
first
method
is
Score
and
Search
E+
approximation
process requires
maximizing
the followwhere
⌃
is
the
covariance
matrix,
µ
is
the
mean,
and
⌦
al Model
(GGM),
and
it
is
assumed
that
the
joint
tion.
SVM
regression
problem,
where
the
solution
to
the
SVM
reour
data
will
produce
non-sparse
GGM
⇢
gression
problem
is
also
the
solution
to
the
Lasso
regression
2
k
k 2
Structure
Learning
gression
problem
is
also
the
solution
to
the
Lasso
regression
(Section
3.1),
and
is
generally
used
to
learn
Bayesian
Neting
posterior
probability:
Model
(GGM),
and
it is assumed
that the joint
future
work,
because
naively
implementing
the
method
on
Arelabi ||2 +Graphical
||xi isz represented
+u
i xi probability
1
i ||2 )1 (11)
nonlinear
dependencies
among
the
random
variables.
Howby
the
following:
T
Bayesian
Parameter
Estimates
Bayesian Parameter Estimates
ity is represented
thev following:
gression
problem
is also
the
solution
to the
Lasso
regression
max(0,
) max(0,
v µ)
)⌦(x
(14) µ))
is
the
inverse
covariance
matrix
or
precision
matrix.
The
2S (v) = by
work
structures.
The
second
approach
is
Constraint
Based
exp(
(x
(6)
While
the
method
presented
by
Fan
L
N
problem.
This
transformation
allows
the
method
to
detect
Variables
105 to 130
Variables 131
to 151
n
1
problem.
This
transformation
allows
the
method
to
detect
where
⌃
is
the
covariance
matrix,
µ
is
the
mean,
and
⌦
⇢N
⇢N
probability
is
represented
by
the
following:
⇧i=1
P
(Y
|X)P
(X)
(3)
utii
our
data
will
produce
non-sparse
models.
ever,
exploring
nonlinear
dependencies
via GGM
this
method
is
2 approach learns a Bayesian
Model
(GGM)
(Section 3.2)
methods,
which
are
used
to learn
Bayesian
and
2 |⌃|
2
k Gaussian Graphical
(2⇡)
problem.
This
transformation
allows
the
method
to
detect
Network
that
is
converted
to
a
+
ū )
(12) T 1
1
, the
nonlinear
dependencies
among
the
random
variables.
Howestimate
the
GGM
that
governs
1
1
nonlinear
dependencies
among
the
random
variables.
HowEstimate
Estimate the join
T
Markov
Networks.
Thework,
final
method
uses
strong
distribution
where
X represents
the
inputs
and
observed
operation schedis the
covariance
matrix
or
precision
matrix.
The
future
because
naively
implementing
the
method
on
Theinverse
soft exp(
thresholding
function
applies
the
Lasso
regression
While
the
method
presented
by
Fan
Li
[16]
can
efficiently
exp(
(x
µ)
⌦(x
µ))
(6)
1
1
nonlinear
dependencies
among
the
random
variables.
How(x
µ)
⌦(x
µ))
(6)
Actual
Actual
T
data
n
1
n
1
assumptions
combined
withand
regression
based
feature
selecule,
and
Yi z,represents
amatrix,
single
output
sample.
Thisthe
apGGM
by
using
regression
coefficients
the
variance
k+1
where
⌃
is
the
covariance
µ
is
the
mean,
and
⌦
ever,
exploring
nonlinear
dependencies
viaE+
this
method
is
sparsity
constraints
over
which
are
incorporated
into
the
1.2
ever,
exploring
nonlinear
dependencies
via
this1.2
method
is
exp(
(x
µ)
⌦(x
µ))
(6)
the
random
variables,
it
is
not
clear
2
2
our
data
will
produce
non-sparse
GGM
models.
z
2 |⌃|
2Bayesian
2 |⌃| 2 learns
n(13) 1
approach
a
Network
that
is
converted
to
a
(2⇡)
(2⇡)
tion
methods
to
determine
dependencies
among
the
random
estimate
the
GGM
that
governs
the
joint
distribution
over
thm
ever,
exploring
nonlinear
dependencies
via
this
method
is
proximation
method
requires
finding multiple
samples that
1
1
2
2
2
(2⇡)
|⌃|
local
subproblem
solutions
on
the
next
optimization
iterais
the
inverse
covariance
matrix
or
precision
matrix.
The
for
each
regression
to
compute
the
precision
matrix
⌦.
The
0.7
0.7
future
work,
because
implementing
the
method
on
future
work,
because
naively
implementing
the
method
on
While
the
method
presented
by
Fan
Li
[16]
can
efficiently
0.8 naively
0.8
variables
(Section
3.3).
maximize
the
posterior
probability,
rather
than
a
single
ason
the
E+
data
set,
which
has
many
com0.6
0.6
GGM
by
using
the
regression
coefficients
and
the
variance
future
work,
because
naively
implementing
the
method
on
subproblems
are solved using ridge retion.
the
E+
random
variables,
not
clear
how
it
will
perform
0.6
0.6 it is
0.5
0.5
approach
learns
a Bayesian
Network
that
is
converted
to
a
⌃ggreis
the
covariance
matrix,
µ
is
the
mean,
and
⌦
signment.
However,
forward
approximating
E+
using
this
where
is
the
covariance
matrix,
u
is
the
mean,
and
is
the
inverse
where
⌃
is
the
covariance
matrix,
µ
is
the
mean,
and
⌦
0.4
0.4
estimate
the
GGM
that
governs
the
joint
distribution
over
our
data
will
produce
GGM
models.
our
data
will
produce
non-sparse
GGM
precision
matrix
is
recovered
by
the
following
computation:
0.4 non-sparse
0.4 models.
where
⌃
is
the
covariance
matrix,
µ
is
the
mean,
and
⌦
al zfor
values
are
computed
by
evaluating
0.3
0.3 to wo
Theregression
key advantage
behind
this particular
Lasso
regression
variables.
The
method
is
intended
our
data
will
produce
non-sparse
GGM
models.
each
to
compute
the
precision
matrix
⌦.
The
method
is
computationally
slower
than
using
statistical
sur0.2
0.2
on
the
E+
data
set,
which
has
many
more
samples
than
GGM
by
using
the
regression
coefficients
and
the
variance
matrix
or
precision
matrix
ation
nverse
covariance
matrix
or
precision
matrix.
The
3.
STRUCTURE
LEARNING
the
E+
random
variables,
it
is
not
clear
how
it
will
perform
iscovariance
the
inverse
covariance
matrix
or
precision
matrix.
The
While
the
method
presented
by
Fan
Li
[16]
can
efficiently
unction
S.
This
function
is
defined
as
formulation
isrogate
that the
main,Therefore,
computationally
demanding,
While
the
method
presented
by
Fan
Li
[16]
can
efficiently
models.
we areor
only
focusing onmatrix.
estimat- The
is
the
inverse
covariance
matrix
precision
104
109
114
119
124
129
130
135
139 that
144 contain
149
154a large n
While
the
method
presented
by
Fan
Li
[16]
can
efficiently
T
1
ple
size
data
sets
precision
matrix
is
recovered
by
the
following
computation:
aster
for
each
regression
to
compute
the
precision
matrix
⌦.
The the
variables.
The
method
is
intended
to
work
with
small
samh learns
aapproach
Bayesian
that
is
converted
to
a
Variables
Variables
on
the
E+
data
set,
which
has
many
more
samples
than
ingaNetwork
building
parameters
andmethod
only
present
this
capability
for
step
is solved
exactly
once.
The
direct
for
comput⌦
=
(1
)
(1
)
(7)
approach
learns
Bayesian
Network
that
is
converted
to
a
estimate
GGM
that
governs
the
joint
distribution
over
learns a Bayesian Network
that
a estimate
the
GGM
that
governs
the
joint
distribution
over
com- Dobra
estimate
the
GGM
that
governs
the
joint
distribution
over
k+1
T
1 is converted to 3.1
Score
and
Search
completeness.
such
as
microarray
data.
While
priors
ar
precision
matrix
is
recovered
by
the
following
computation:
ing
x
requires
computing
the
matrix
(A
A
+
⇢I)
.
The
et.
al
(2004)
and
Li
et.
al
(2006)
variables.
The
method
is
intended
to
work
with
small
samy
using
the
the
variance
i regression coefficients and
T
1
ple
size
data
sets
that
contain
a
large
number
of
features,
(e)
(f)
the E+ Score
random
variables,
it variables,
is not
clear
how
itclear
will
perform
by using
the
regression
coefficients
and
the
variance
y.
GGM
by
using
the
regression
coefficients
and
the
variance
and
Search
(S&S)
algorithms
try
to
search
through
, v InGGM
) resulting
max(0,
v
)
(14)
the
E+
random
variables,
it
is
not
clear
how
it
will
perform
⌦
=
(1
)
(1
)
(7)
the
E+
random
it
is
not
how
it
will
perform
matrix
never
changes
throughout
the
entire
optiTmatrix
1
where
represents
an
upper
triangular
matrix
with
zero
difor
problems
with
more
features
than
o
⇢N
⇢N PROBABILISTIC
ple
size
data
sets
that
contain
a
large
number
of
features,
to
compute
the
precision
⌦.
The
the
space
of
all
possible
models
and
select
the
best
seen
e regression
Al-for
such
as
microarray
data.
While
priors
are
generally
reserved
2.
GRAPHICAL
MODEL
on
the
E+
data
set,
which
has
many
more
samples
than
=
(1
)allows
(1
) matrix
(7) on on
forregression
each
regression
to
compute
thethe
precision
⌦. The
each
to ⌦compute
the
precision
matrix
⌦. The
mization
process.
Storing
this
result
distributed
the
E+
data
set,
which
has
many
more
samples
than
the
E+
data
set,
which
has
many
more
samples
than
Figure
2.
Bayesian
GGM’s
parameter
estimates
compared
against
the
actual
model.
The
best
model
is
selected
by
using
a
global
critesuch
as
microarray
data.
While
priors
are
generally
reserved
n
matrix
is
recovered
by
the
following
computation:
agonals
the
non-zero
elements
represent
the
regression
(e.g.,
microarray
data
sets),
our
E+
data
Learning
the
true
joint
probability
distribution
directly
in and
gover
function
applies
the Lasso
regression
optimization
method
to
perform
aby
very
computationally
in- diagonals
variables.
The
method
is
intended
to
work
with
small
samwhere
represents
an
upper
triangular
matrix
with and
zero
the
where
represents
an
upper
triangular
matrix
with
zero
difor
problems
with
more
features
than
observation
samples
precision
matrix
is
recovered
by
the
following
computation:
precision
matrix
is
recovered
the
following
computation:
values
on
300
randomly
sampled
MO2
simulations.
ria function,
such parameter
as The
theThe
likelihood
function.
However,
the
variables.
method
is
intended
to
work
with
small
samvariables.
method
is
intended
to
work
with
small
samsver
be-z, which
k+1
the
general
case
is
computationally
intractable,
especially
1
where
represents
an
upper
triangular
matrix
with
zero
difor
problems
with
more
features
than
observation
samples
are
incorporated
into
the
tensive
task
once
and
reduce
all
future
x
computations
most
common
criteria
function
is the a
Bayesian
Information
i coefficients,
and
represents
a
diagonal
matrix
containas
well.
Our
E+
simulations
generate
non-zeroand
elements
represent
theaelements
regression
coefficients,
and
represents
Tnon-zero
1 with
ple
size
data
sets
that
contain
large
number
of
features,
agonals
the
represent
the
regression
(e.g.,
microarray
data
sets),
our
E+
data
set
requires
priors
when
dealing
large
number
of
continuous
random
T
1
eutions
folple
size
data
sets
that
contain
a
large
number
of
features,
k+1
⌦
=
(1
)
(1
)
(7)
T
1
on agonals
the
next
optimization
iteraple
size
data
sets
that
contain
a
large
number
of
features,
andvariables.
the
non-zero
elements
represent
(e.g.,
steps. Caching
the
values
used
to
compute
xthe
to)probability
disk the regression
Criteria (BIC)
[22]: microarray data sets), our E+ data set requires priors
⌦
=
(1
)
(1
(7)
ieach
1
In
general,
it
is
assumed
joint
fac⌦
=
(1
)
(1
)
(7)
a
diagonal
matrix
containing
the
variances
for
performed
regression.
such
as microarray
data.
WhileOur
priors
are
generally
reserved
a reing
the
variances
for
each
performed
regression.
There
are
vectors
per
input
parameter
set,
which
le
1represents
such
as
microarray
data.
While
priors
are
generally
reserved
coefficients,
and
a
diagonal
matrix
containas
well.
E+
simulations
generate
30,540
output
data
allows
a
2.2GHz
Intel
Core
i7
laptop
to
solve
a
univariate
coefficients,torizes
and into several
represents
diagonalcomputational
matrix contain- such as
well. Our E+
simulations
generate
30,540reserved
output data
as microarray
data.
While priors
are generally
smaller moreamanageable
log
M
,behind
the
this3.9GB
particular
Lasso
regression
represents
an
upper
triangular
matrix
with
zero
difor
problems
with
more
features
than
observation
samples
Lasso
regression
problem
in
approximately
17
min⇤
Dim[G]
(4)
BIC(Data;
G)
=
L(Data;
G,
✓)
+
Pros:
Scalable
for
a
large
number
of
random
variables
where
represents
an
upper
triangular
matrix
with
zero
difor
problems
with
more
features
than
observation
samples
problems.
Factorization
is
based
purely
on
conditional
inmethods
for
learning
⌦
from
the
data
directly,
either
by
imbalance
between
output
and
input
ob
ing
the
variances
for
each
performed
regression.
There
are
vectors
per
input
parameter
set,
which
leads
to
a
significant
ing
the
variances
for
each
performed
regression.
There
are
vectors
per
input
parameter
set,
which
leads
to
a
significant
2
where
represents
an
upper
triangular
matrix
with
zero
difor
problems
with
more
features
than
observation
samples
ight.
he
main,
computationally
demanding,
utes.
dependence
and
is
represented
using
a
graphical
model
G,
1
and
themethods
non-zero
elements
represent
the
regression
(e.g.,
microarray
data
sets),
our
E+
data
set
requires
priors
agonals
and
the
non-zero
elements
represent
the
regression
(e.g.,
microarray
data
sets),
our
E+
data
set
requires
priors
for
learning
⌦
from
the
data
directly,
either
by
imbalance
between
output
and
input
observations.
For ex- produce
estimating
⌃
and
computing
⌃
.
The
inverse
operation
ample,
299
E+
simulations
appr
robmethods
for
learning
⌦
from
the
data
directly,
either
by
imbalance
between
output
and
input
observations.
For
exProblem
for
the
E+
Data
where
Dim[G]
represents
the
number
of
parameters
estionce.
The direct
for
computwhere
G is aSet
graph
with
V
nodes
representing
random
variagonals
and
the
non-zero
elements
represent
the
regression
(e.g.,
microarray
data
sets),
our
E+
data
set
requires
priors
1 method
1
11diagonal
nts,
and
represents
matrix
containas
well.
Our
E+
simulations
generate
30,540
output
data
T
1 aE diagonal
3operation
coefficients,
and
represents
a
matrix
containas
well.
Our
E+
simulations
generate
30,540
output data
slave
mated
within
the
graph,
M
is
the
total
number
of
samples,
ables
and
edges.
The
edges
in
E
represent
conditional
estimating
⌃
and
computing
⌃
.
The
inverse
ample,
299
E+
simulations
produce
approximately
3.9GB
of
1
1
mputing
the
matrix
(A
A
+
⇢I)
.
The
4.2
Bayesian
Regression
GGM
Learning
is
a
user
defined
hyperparameter.
Using
the
above
prior
final
estimate
is
computed
using
the
following
equation:
requires
an
O(V
)
runtime,
where
V
is
the
total
number
data.
In
addition,
the
simulation
input
p
is
a
user
defined
hyperparameter.
Using
the
above
prior
final
estimate
is
computed
using
the
following
equation:
estimating
⌃
and
computing
⌃
.
The
inverse
operation
ample,
299
E+
simulations
produce
approximately
3.9GB
of
coefficients,
and
represents
a
diagonal
matrix
containas
well.
Our
E+
simulations
generate
30,540
output
data
i
3optiandper
L is input
the
the log-likelihood
function.
dependencies
between
the
random
variables
in
the total
graph
G.number
and some
additional
derivations,
Fan Li [16]There
derived
the folvariances
for
each
performed
regression.
are
vectors
parameter
set,
which
leads
to
a
significant
er
changes
throughout
the
entire
ing
the
variances
for
each
performed
regression.
There
are
vectors
per
input
parameter
set,
which
leads
to
a
significant
E+
Building
Parameters
X
requires
an
O(V
)
runtime,
where
V
is
the
data.
In
addition,
the
simulation
input
parameter
space
repand some additional
Fan
Li
[16]
derived
the
fol3
X
Unlikederivations,
the direct
methods,
the
Bayesian
approach
assumes
1
lowing
maximum
a
posteriori
(MAP)
distributions
for
all
M
are
used toprobfind
the
best
ˆS&S
The
graph
structure
is intended
to
represent
thethe
true
factorof
random
variables,
which
is
not
scalable
to
larger
resents
a
complex
high
dimensional
stat
requires
an
O(V
)
runtime,
where
V
is
total
number
data.
In
addition,
the
simulation
input
parameter
space
repX
= These
P (✓ methods
) per
(19)generally
ing
the
variances
for
each
performed
regression.
There
are
vectors
input
parameter
set,
which
leads
to
a
significant
1
1
oring
this
result
allows
the
distributed
lowing
maximum
a
posteriori
(MAP)
distributions
for
all
1either
and
allthe
: data
M
s for learning
⌦
from
directly,
either
by
imbalance
between
output
and
input
observations.
For
exSample
ID
Pfor
P ⌦
… from
P matrix.
a
global
Wishart
prior
the
precision
Using
this
methods
for
learning
the
data
directly,
by
imbalance
between
output
and
input
observations.
ex- furˆ
of
random
variables,
which
is
not
scalable
to
larger
probresents
a
complex
high
dimensional
state
space,For
which
structure
for
Bayesian
Networks,
because
the
log-likelihood
=
P
(✓
)
(19)
ization
for
the
joint
probability
distribution
by
splitting
it
j
j
1
i
generate
and
all random
:a veryfor
One Observed
Output
Y
Mby
1
to
perform
computationally
in1
lems.
More
importantly,
directly
learning
⌦
via
[7]
only
ther
complicates
the
problem.
The
inpu
of
variables,
which
is
not
scalable
to
larger
probresents
a
complex
high
dimensional
state
space,
which
fur1 learning
Min(V
)
avg
avg
methods
⌦
from
the
data
directly,
either
imbalance
between
output
and
input
observations.
For
exglobal
prior
over
the
precision
matrix
allowed
Fan
Li
[16]
function
factorizes
into
a
series
of
summations.
This
factorj=1
into
simpler
components.
These
types
of
models
have
been
P
(
|
,
D)
/
ng
⌃
and
computing
⌃
.
The
inverse
operation
ample,
299
E+
simulations
produce
approximately
3.9GB
of
estimating
⌃
and
computing
⌃
.
The
inverse
operation
ample,
299
E+
simulations
produce
approximately
3.9GB
of
lems. xMore
importantly,
directly
learning
⌦where
via
only
the problem. The input parameter space
k+1
M is [7]
the total
number of samplesther
and ˆ complicates
is a sample
ation
P
P
P
p
d reduce alltofuture
computations
Row
ID
V
V
…
V
Max(V
) avg
avg x1estimates
ization
makes
very between
easy
to modify
an
existing Bayesian and observational skewness has lead us
i2
prove
that
the
Bayesian
approach
ainverse
e↵ectively
applied
to
many
fields,
such
asglobally
topic
3
3
computed
using
Eq.
18.
Afterward
a few1it
iterations
(x
) +guarantees
| | classification
a
global
maximum
solution
if
there
are
sufficient
estimating
⌃
and
computing
⌃
.
The
operation
ample,
299
E+
simulations
produce
approximately
3.9GB
of
lems.
More
importantly,
directly
learning
⌦
via
[7]
only
ther
complicates
the
problem.
The
input
parameter
space
k+1
P
(
|
,
D)
/
i guarantees
a
global
maximum
solution
if
there
are
sufficient
and
observational
skewness
has
lead
us
to
avoid
exploring
ˆfinal
an i O(V
)
runtime,
where
V
is
the
total
number
data.
In
addition,
the
simulation
input
parameter
space
reprobrequires
an
O(V
)
runtime,
where
V
is
the
total
number
data.
In
addition,
the
simulation
input
parameter
space
repexp(
)
where
M
is
the
total
number
of
samples
and
is
a
sample
1
values
used
to
compute
x
to
disk
estimating
s
and
s,
we
use
the
estimates
to
comi
Network
and
compute
the
modified
score
without
recom…
…
[2],
classification
[1], and
disease
diagnosis [13].
i 3document
precision
matrix
possible
variable
orders.
PDoptimal P
Pall
pover
N
N
2
pute
the
GGM’s
precision
(Eq.
7).In score.
withcomputed
using
Eq.
18.
Afterward
a matrix
few
iterations
between
(x
x
)
+
|
|
(16)
puting
the
entire
For
example,
adding
anthe
edge
to an
requires
an
O(V
)
runtime,
where
V
is
the
total
number
data.
addition,
the
simulation
input
parameter
space
repfor
the
problems
However,
the
direct
approaches
such
as
[7],
which
will
most
likely
only
2probnirandom
ij
njavg
iwhich
ijdimensionality.
l Core
i7 observations
laptop
to
solve
a
univariate
The
graphs
used
to
represent
factorized
joint
probabilguarantees
a
global
maximum
solution
if
there
are
sufficient
and
observational
skewness
has
lead
us
to
avoid
exploring
observations
for
the
problems
dimensionality.
However,
direct
approaches
such
as
[7],
which
w
of
variables,
is
not
scalable
to
larger
probresents
a
complex
high
dimensional
state
space,
which
furom
variables,
which
is
not
scalable
to
larger
resents
a
complex
high
dimensional
state
space,
which
fur299
avg
Max(V
)
n=1
j=i+1
j=i+1
(a)
(b)
That
is
to
say,
under
the
Bayesian
formulation
presented
1
exp(
)
estimating
s and
s,orwe use existing
the final Bayesian
estimates to
com- requires computing the new and
mizanetwork
ity
distributions
are
either
direct
acyclic
(DAG)
… graphs
on
problem
in
approximately
17
mini
5.precision
RESULTS
bylems.
Fan Livariables,
[16],
allthe
variable
orderings
should
More
importantly,
directly
learning
⌦
via
[7]
only
ther
complicates
the
problem.
The
input
parameter
space
of
random
is
not
scalable
to
larger
probresents
a
complex
high
dimensional
state
space,
which
furHow
to directly
sample
Xwhich
to
cover
the
whole
space
oftheoretically
More
importantly,
learning
⌦
via
[7]
only
ther
complicates
the
problem.
The
input
parameter
space
observations
for
problems
dimensionality.
However,
the
direct
approaches
such
as
[7],
which
will
most likely only
pute
the
GGM’s
matrix
(Eq.
7).
previous
conditional
probability
involving
the
child
of
the
P(
|✓ , graphs.
, D) / P (D|The, first
,(16)
✓ )P (form
|
, ✓ )P (35,040
|✓that
)
undirected
assumes
theIn graph
tting
Figure
order to assess how well the GGM estimates
building3. Figure 3(a) highlights that building parameter three has values that
building
parameters
for
all
possible
real
values?
produce
the
same
precision
matrix.
guarantees
a global
maximum
if ⌦
there
are
observational
skewness
lead
usexploring
to
avoid exploring
new edge,
and
adding
to
the previous
BIC
represents
a Bayesian
Network
and sufficient
the latter
form
assumes
+ 1directly
+if
N there
2i + |D|solution
ees
a global
maximum
solution
are
and
observational
skewness
has
lead
ushas
to
avoid
More
importantly,
learning
viaparameters,
[7] sufficient
only
ther
complicates
the
problem.
The
input
parameter
space
we experimented
withand
three
data
sets –the
Finedi↵erence
opti-lems.
⇠
Gamma(
,
occasionally
differ
greatly from
the parameter’s mean, and Figure 3(b) compares
The Wishartthat
prior
used
by Fan
Li
defined
as W
( , TThese
). Graingraph
(FG), Model score.
Order 1 (MO1),
and Model
Order
2
2 [16]aisMarkov
Updating
the
penalty
term
is
achieved
by
simply
the
graph
represents
Network.
5.
RESULTS
Pdimensionality.
P 1 dimensionality.
P
mmuobservations
for
the
problems
However,
the
direct
approaches
such
as
[7],
which
will
most
likely
only
egression
GGM
Learning
1for the
1
1
tions
problems
However,
the
direct
approaches
such
as
[7],
which
will
most
likely
only
log
M
(MO2).
The
first
data
set
contains
building
simulations
guarantees
a
global
maximum
solution
if
there
are
sufficient
and
observational
skewness
has
lead
us
to
avoid
exploring
✓
+
✓
+
(x
x
)
parameters
estimated
via
a
genetic
algorithm
and
gradient
optimization.
P ( i |✓i , i ,represents
D) / P (D| aitypes
, i , ✓both
, ✓ithat
)P ( ia joint
|✓i ) probability
user
defined
and In
T order
is a)distribution
hyadding N 2 to the updated BIC score, where N is the
i )P ( assume
i | ihyperparameter
facto
assess
how
well
the
GGM
estimates
building
built from
small
brute
force
increments
across
14 building
2
ethods,
the
Bayesian
approach
assumes
Bayesian
Regression
GGM
Learning
number
ofdata
newly
estimated
torizes
according
to their
structure.
However,
a parameters.
Bayesian
perparameter
diagonal
matrix who’s
entries
are parameters,
governed
by
The MO1
data
set
contains
299
building
simu-parameters.
observations
for
the
problems
dimensionality.
However,
the
direct
approaches
such
as
[7],
which
will
most
likely
only
+
1
+
N
2i
+
|D|
we
experimented
with
three
sets
–
Fine
(17)
sub-⇠ Gamma(
,
Random Parameter Estimation
0.8
0.6
0.4
0.2
0.6
0.4
0.2
27 28 29 30 66 67 68 69 157 158 159 160 181
Variables
Parameter Values
Parameter Values
Parameter Estimates
Parameter Estimates
Parameter Estimates
Parameter Values
Parameter Values
2
0.8
9
Parameter Values
Parameter Estimates
1
1
0
27 28 29 30 66 67 68 69 157 158 159 160 181
Variables
Parameter Estimates
⇢N
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
9
i
Estimate
Actual
Parameter Values
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
Parameter Estimates
Estimate
Actual
Parameter Values
Parameter Estimates
Bayesian Parameter Estimation
N
Parameter Values
Parameter Estimates
⇢N
MO2 Building Parameter 3
FG Building Parameter 2
1
i
M
1
1
1
i
2
D P1
n=1
ni
ij
nj
1.5
1
0.5
j
1
N
j=i+1
One−Act
Zero−Act
Est
Zero−Est
Aactual
Gradient−Est
GA−Est
1
0.5
0
j=1
P1
i
j
i
156
1
1.5
Parameter Values
Parameter Value
2
2
N
j=i+1
i
1
ij
2
1
i
90
1
0
0
50
100
150
Simulation
200
250
300
−0.5
0
5
10
15
20
25
30
Simulations
35
40
45
50
i
P156
1
i
i
N
j=i+1
1
i
i
1
2
ij i
1
i
i
D
n=1
i
i
ni
1
i
1
i
N
j=i+1
i
j
nj
i
2
The
most
common
method
for performing S&S within
lations and ⇠150 building
parameters.
The
building paramor for the precision
matrix.
Using assumes
this
Network
a much simpler probabilistic
dependency
the following
Grain
(FG),
Model
Order
1
(MO1),
and
Model
Order
2
2 distribution:
Maximizing P
i | i , D) with respect to i is equivalent to
eters were independently
set
to their minimum
or maximum
s precision
[3].PN
P
P(variables,
the
literature
is
Greedy
Hill Climbing, which explores the
between
the
in
which
a
variable
is
conditionally
D
N
matrix
Fan
Li
[16]
1 allowed
1
2
2
(MO2).
The
first
data
set
contains
building
simulations
solving
a
Lasso
regression
problem
with
the
regularization
value,
while
other
parameters
were
set
to
their
average
value.
Hyper-+ ✓i + independent
jx
pnj )
j=i+1 ij ✓i
n=1 (xni
j=i+1
candidate
graph
space
byThe
deleting edges, adding edges, and
fined
✓i i[16].
of
all
other
variables
givenauthors
its small
parents.
On
hyperparameter
set to
The original
This
results
in two
simulations
per
building
parameter.
)
ayesian approach
estimates
a
globally
built
from
brute
force
increments
across
14
building
P (✓these
) =formulations
exp(
) with microarrays,
(15) MO2Xdata
ihand,
parameter θi the
2derived
to work
which
changing
directions.
The best valid graph modification
set contains
pairwise edge
minimum
and maximum
other
a
Markov
network
assumes
variables
are
2
2
parameters.
The
MO1
data
set
contains
299
building
simuatrix over all possible variable
orders.
(17)
typically contain 10,000 to 12,000
variables, but have very
•  Our
FG
and function,
MO2 experimental
results indicated that the GGM performs
combinations
across the
same building
parameter
set.
is
selected
according
to
a
criteria
generally
BIC,
conditionally
independent
from
variables
Y
provided
that
for
T
1
and ⇠150 building
parameters.
Theabuilding
few
samples. This allowed the authors to uselations
conventional
Using the FG
data set, we built
GGM usingparam250 simular (9)
the Bayesian
formulation
presented
This
version
assumes
we
are
only
splitting
the
optimization
and
a
new
candidate
graph
is generated. A
modification
is
they
are
separated
by
variables
Z.
X
and
Y
are
said
to
be
well
at
estimating
building
parameters.
methods
to solve forto i . However,
the
E+independently
Maximizing P ( i | i , D) with optimization
respect to
tions sampled
without
replacement,
evaluated
the model
1 and or
i is equivalent
eters
were
set
to
their
minimum
maximum
is in
aZthis
user
defined
hyperparameter.
Using
theX
above
prior
final
estimate
is
computed
using the
following
equation:
1
ariable orderings
should
theoretically
problem
across
the
training
samples,
and
not
the
features.
i prior
only
valid
if
the
candidate
graph
is
a
valid
Bayesian
Net- equation:
data
set
used
work
contains
several
million
data
vecseparated
by
if
and
only
if
all
paths
between
and
Y
using
150
test
simulations
simulations.
In
addition,
we
built
is
a
user
defined
hyperparameter.
Using
the
above
final
estimate
is
computed
using the following
solving
a
Lasso
regression
problem
with
the
regularization
i
value,
while
other
parameters
were
set
to
their
average
value.
and
some
additional
derivations,
FanADMM
Liap[16] derived
the
fol-the entire
con- matrix.
p
tors,
which
mostly
invalidates
conventional
optimization
It
is
possible
to
split
across
both.
[3]
presents
the
aNetwork
GGM
using
MO1the
data
set.approach
The resulting
model
ecision
work.
This
guarantees
a locally optimal solution
M
and
some
additional
derivations,
Fan
Li
[16] derived
folpass
through
the
variables
Z
[14].
The
Bayesian
2 in two simulations
X
hyperparameter
setprior
to
[16].
The
original
authors
Wishart
i
This
results
per
building
parameter.
The
M
1
1
proaches.
Rather
than
using
the
fast
grafting
algorithm
lowing
maximum
a
posteriori
(MAP)
distributions
for
all
1
is evaluated using 300 sampled MO2 simulations.
1
X
blem
formulation
for
supporting
this
functionality.
ˆ
1(19)
1
is
a
user
defined
hyperparameter.
Using
the
above
prior
final
estimate
is
computed
using
the
following
equation:
that
will
maximize
the
criteria
function,
but
could
be farbuilt
=
P
(✓
)
lowing
maximum
a
posteriori
(MAP)
distributions
for
all
makes
stronger
assumptions
about
independence,
but
these
•
the
Bayesian
models
using
the
FG
and
full
MO1
data
sets
1j
used
by Fan
Li
[16]
is
defined
as
W
(
,
T
).
j
1 the Lasso
ii Overall,
ˆ
derived
these
formulations
to
work
with
microarrays,
which
used
by
[16],
we
solve
regression
problems
using
Analyzing
Figure
1
illustrates
that
the
building
paramMO2
data
set
contains
pairwise
minimum
and
maximum
=
P
(✓
)
(19)
and all
:
M
for Ω
j
j
1
i
j=1
Solved
by all
and
:
and
some
additional
derivations,
Fan
Li
[16]
derived
the
folaway
from
the
true
model.
M j=1
ADMM
(Section
4.1).
assumptions
make
it
easier
to
perform
exact
interference.
eter
estimates
using
the
GGM
are
mostly
better
than
the
efined
hyperparameter
and
T
is
a
hytypically contain 10,000 to 12,000 variables, but have very
combinations
across
the
same
building
parameter
set.
M
are
statistically
better
than
the
uniform
distribution,
which
is
expected.
X
LASSO
with
After
obtaining
’s
MAP
estimate,
it
can
be
used
to
i
estimates
generated
by
randomly
guessing
using
a
[0,
1]
ran1
1
There
are
two
algorithms
that
extend
the
basic
algorithm
lowing
maximum
a posteriori
(MAP)
distributions
for all
contrast,
Markov
Networks
use approximate
inference
1
1
ˆ
samples.
This
allowed
the
authors
conventional
PP(by
/
alfew
matrix
who’s
entries
areIn
governed
Using
the
FG
data
set,
we
built
a
GGM
using
250
simulai | 1use
i|✓, iD)
ˆ
=
(✓j )
(19)
ADMM
maximize
(to
,
,
D)
with
respect
to
.
However,
jis aP
i
1
dom distribution. The
overall
error rate
where
MGGM’s
is the absolute
total
number
of
samples
and allows
sample
1 P ( i | i , D) /i
i
i
i
by
constraining
the
search
space,
which
for
better
soˆ
and
all
:
1
methods,
such
as
Belief
PDLoopy
PN
P
p✓i Additionally,
M
where
M is the total
number
of samples
and i is a sample
optimization methods to solve
. However,
the
E+
N
P ( for
upon
the Propagation.
hyperparameter
2
tions
sampled
without
replacement,
and
evaluated
the
model
is
statistically
better
with
95%
confidence
than
the
uniform
i |✓i , i i , D) is dependent
tion:
j=1
computed
using
Eq.
18.
Afterward
a
few
iterations
between
P
P
P
p
(x
x
)
+
|
|
ni
ij
nj
i
ij
N
N
Maximum a posterior
n=1
j=i+1D
j=i+1 2
lutions.
The first algorithm is the Sparse Candidate method
Conclusions
both
have
weaknesses
certain
con- error
andgraphs
Fan
Li exp(
[16]
noted
that
there
is with
not a representing
known
computed
using
Eq.
18.
Afterward
a few iterations
between
distribution’s
rate
– 4.05±0.98
However,
1built
data set used in this work contains
several
million
data
vec(xni method
| addition,
)
ij xnj ) + simulations.
i estimating
ij |vss 5.07±1.01.
using
150 test
simulations
we
•
However,
our
current
GGMs
have
difficulty estimating parameters that
and
s,
we
use
the
final
estimates
to
comn=1
j=i+1
j=i+1In
1 variables that have1 a
to
analytically
remove
the
hyperparameter
via
integration.
only
the
error
rates
for
variables
1,
2,
3,
8,
10,
11,
12,
13,
exp(
)
[8].
This
method
assumes
that
random
i
ditional
independence
properties.
However
within
this
work,
P (optimization
/ apestimating
s and
s,ofwe
use theand
finalˆestimates
to comi | i , D)
distribution
for
✓iinvalidates
tors, which(MAP)
mostly
conventional
a
GGM
using
the
entire
MO1
data
set.
The
resulting
model
where
M
is
the
total
number
samples
is
a
sample
pute
the
GGM’s
precision
matrix
(Eq.
7).
i
This means
numerical methods
are required
to approximate
and
14i are
statistically
better
than their
random
guessing
(16)
deviate
significantly
from
the
mean.
This
implies
we
need
to
explore
(✓proaches.
exp(
)
(15)
2 Networks
P
P
P
p
high
measure
of
mutual
information
should
be
located
closer
i) =
we
show
preference
to
Markov
over
Bayesian
Netpute
the
GGM’s
precision
matrix
(Eq.
7).
-1
D
N
N
2
Rather
than
using
the
fast
grafting
algorithm
many
nu- using 300 isampled
isareevaluated
MO2
simulations.
(16)
The other
arecomputed
not statistically
di↵erusing
Eq. 18. Afterward a few iterations between
2 all β2and all ψ the integral over the hyperparameter.
(xni Therej=i+1
|toijvariables
|each
ij xnj ) +counterpart.
n=1
j=i+1
in the
final network
than variables with
low
because
wish to using
avoid
falsely
representing
variables
1
merical
methodswe
for
computing
approximates
toAnalyzing
definite in- Figure
ent.
used by [16], we solve the works
Lasso
regression
problems
different
hyper-parameter
settings,
which
may
induce
more
estimation
exp(
) RESULTS
1 illustrates 5.
that
theother
building
paramestimating
s
and
s,
we
use
the
final
estimates
to
comtegrals,
such as1 Trapezoidal
method and 1Simpson’s Rule.i 1
mutual
The
mutual information for two dis1
While the
other variables
areinformation.
not statistically di↵erent,
s ADMM
we are only
splitting
optimization
conditionally
independent.
(Section
4.1). theas
5.
RESULTS
P
(
|✓
,
,
D)
/
P
(D|
,
,
✓
)P
(
|
,
✓
)P
(
|✓
)
eter
estimates
using
the
GGM
are
mostly
better
than
the
i
i
i
i
i
i
i
pute
themodel
GGM’s
precision
matrix
(Eq.
7).
i
However, the integral
over ✓i is unbounded
1 i from above, be- i
1there is inot a clear 1indication
1 the
variance,
or
possibly
a
mixture
model
approach,
which
will
allow
more
that
GGM
fails
to
In
order
to
assess
how
well
the
GGM
estimates
building
(16)
crete
random
variables,
X
and
Y
,
is
defined
as
follows:
raining
samples,
and
not
the
features.
Maximize
While
these
graphical
models
are
able
to
adequately
repP
(
|✓
,
,
D)
/
P
(D|
,
,
✓
)P
(
|
,
✓
)P
(
|✓
)
i
i
i
i
i
i
i
After obtaining i ’s MAP cause
estimate,
it can
used
to
i
i represent
i
estimates
generated
by randomly
guessing
using
aexperimented
[0,
ran✓i ’s values
exist be
in the
interval
[0,1),
which
means
these variables.
This iindicates
that
the1]
GGM
over-to
In
order
assess
how
wellsets
the–GGM
estimates building
+
1
+
N
2i
+
|D|
parameters,
we
with
three
data
Fine
1
across
both.
the
✓
◆
resent
joint
graphintegral
structures
generally
⇠
Gamma(
,for
variable
means.
Eq. ADMM
17
mustdistributions,
beto
transformed
tothe
a bounded
theNare2i
maximize
P ( i[3]|✓presents
with
respect
However,
all
isThe
successfully
modeling
the absolute
FG buildingerror
parameters.
In we
i , i , D) with
i.
+
1
+
+
|D|
dom
distribution.
overall
GGM’s
rate
X
parameters,
experimented
with
three
Grain
(FG),
Model
Order
1 (MO1), andPModel
Order
2 data sets – Fine
2
(x,
y)
1
⇠
Gamma(
,
numerical
methods
to
be
applicable.
Given,
Eq
17’s
com5.
RESULTS
orting
this
functionality.
fact
analyzing
Figures
1(a)
indicates
that
the
GGM
isolates
Given
total
variables
PDis statistically
PN2 better
I(X,
Y Grain
)uniform
= set
P contains
(x,Model
y)logbuilding
(5)
P ( i |✓i , respect
upon theP
hyperparameter
✓i1 number
1 2 the
1of random
12
1(MO2). than
i , D) is dependent
with
95%
confidence
the
N
1
(FG),
Order
1
(MO1),
and
Model Order 2
to ψ-1predetermined.
The
first
data
simulations
plex nature and
Li’s
[16]
recommendation
to
use
sampling
✓
+
✓
+
(x
x
)
P
(
|✓
,
,
D)
/
P
(D|
,
,
✓
)P
(
|
,
✓
)P
(
|✓
)
the
building
parameter’s
means
very
well.
While
the
uniP
(x)P
(y)
ni
j
nj
i
i
i
i
i
i
i
ij
i1
i it is PN i
P
PD rate
i
i N
n=1
an
E+j=i+1
for
our
experiments)
1 j=i+1
21 |✓ i, 1 , D]simulation,
2
In variances
order
to
assess
how data
well
the14
GGM
estimates
building
x,y
and Fan Li [16] noted thatwithin
there
is not
a(⇠300
known
method
distribution’s
error
–
4.05±0.98
vs
5.07±1.01.
However,
(MO2).
The
first
set
contains
building
simulations
)
to approximate
,
we
elected
to
use
E[
as
built
from
small
brute
force
increments
across
building
i
i
form
distribution
matches
the
mean
and
(Figure
✓
+
✓
+
(x
x
)
i
i
ni
j
nj
i
j=i+1 2 ij i
n=1
j=i+1
1
+
1
+
N
2i
+
|D|
our estimates for via
,
which
is
the
MAP
estimate
under
a
to analytically remove the hyperparameter
integration.
parameters,
wethe
experimented
with
three
data
sets14–building
Fine
1(b)), for
it appears
the matching
only
possible
because
built
from
small
brute299
force
increments
across
only the error, rates
variables
1, 2, 3,is )8,
10,
11,
12,
13,
i
parameters.
The
MO1
data
set
contains
building
simu⇠
Gamma(
(17)
2
distribution.
Based on Fan Li’s [16] MAP
distribuactual
building
parameters
have
arandom
large
distribution
This means numerical methodsGamma
are required
to approximate
Grain
(FG), range,
Model
1building
(MO1),
and Model
Ordersimu2
2
parameters.
TheOrder
MO1
set contains
299 building
and
14
are
statistically
better
than
their
guessing
lations
and
⇠150
building
parameters.
Thedata
param(17)
tion, we derived the
following
maximum
likelihood
estimate
which
exPare
PDto i is equivalent
PNmostlytocenters 2at 0.5, the uniform distribution’s
N 1Pmany
1 with 1counterpart.
2 i ,nuMaximizing
(
|
D)
respect
the integral over the hyperparameter.
There
-1
-1
lations
and
building
parameters.
The building
parami
eters
were
independently
set
to⇠150
their
minimum
or maximum
(MO2).
The
first
data
set contains
building
simulations
The
other
variables
are
not
statistically
di↵er1.  (MLE)
Given aequation
fixed θ ,for
we j=i+1
estimated
the
+ ✓i by+computing
value. j xnj )
ija✓iψ sample
i :
This
work
was
funded
by
field
work
proposal
CEBT105 under the Department
n=1 (xni pected
j=i+1
Maximizing
P
(
|
,
D)
with
respect
to
is
equivalent
to
solving
a to
Lasso
regression
with
the
regularization
i
i
i
merical methods for computing
approximates
definite
in- pproblem
eters
were
independently
set
to
their
minimum
or
maximum
value,
while
other
parameters
were
set
to
their
average
value.
)
maximum
likelihood
estimate
(MLE)
ent.
built
small
Similar to the FG data set results,
the from
Bayesian
GGMbrute force increments across 14 building
2
1 +solving
N
hyperparameter
set 2i
to+a|D|
[16].
The
original
authors
Lasso
regression
problem
with the
regularization
of
Energy
Building
Technology
Activity
Number BT0305000. This research
1
presents
excellent
performance
on the
MO2
set
(Figure
i While
This
results
in
twodata
simulations
per
building
parameter.
The
value,
while
other
parameters
were set
to their
average
value.
tegrals, such as Trapezoidal method
and
Simpson’s
Rule.
the
other
variables
are
not
statistically
di↵erent,
parameters.
The
MO1
data
set
contains
299
building
simup
=
P
P
P
i
N
D
N
(17)
1
2
2).i The
Bayesian
method
isauthors
better set
thancontains
a uniform
distri- in two
✓i 1 + above,
✓formulations
+hyperparameter
xnjmicroarrays,
)2
derived
these
with
which
set
to
[16].
The original
ij
MO2
data
pairwise
minimum
and
maximum
i
This
results
simulations
per
building
parameter.
The
j=i+1 ijfrom
n=1 (xni to work
j=i+1
However, the integral over ✓i is unbounded
bethere is not a clear
indication
that
the
GGM
model
fails
to
lations
and
⇠150
building
parameters.
The
building
paramused
resources
of
the
Oak
Ridge
Leadership
Computing
Facility
at
the
Oak
bution
at
estimating
all
building
parameters.
The
Bayesian
(18)
typically
contain
10,000
to
12,000
variables,
but
have
very
derived
these
formulations
to
work
with
microarrays,
which
combinations
across
the
same
parameter
set.minimum and maximum
data building
set contains
pairwise
cause ✓i ’s values exist in the
interval
[0,1),
which
means
2.  Computed
multiple
samples
and
E[ψ-1respect
|θi,these
βi, D] variables.
1 MLE
1 represent
Maximizing
P
(
with
to
is
equivalent
to
This
indicates
the
GGM
overGGMi has
better
overall
error
—that
13.01±0.85
vsMO2
41.6936±2.13.
i |a estimated
i , D)
eters
were
independently
set
to
their
minimum
or is
maximum
Given few
a fixed
✓
,
we
can
estimate
sample
by
comi
i
Ridge
National
Laboratory,
which
by the Office of Science of the
samples.
This
allowed
the
authors
to
use
conventional
typically
contain
10,000
towith
12,000
variables,
butproduces
have very
Using
the
FG data
set, predicwe builtacross
a GGM
using
250
simula-parameter supported
using
weighted
sampling
combinations
the
same
building
set.
In
addition,
the
model
statistically
better
1
Eq.
17
must
be
transformed
to
a
bounded
integral
for
the
puting
the
maximum
likelihood
estimate
(MLE)
(Eq.
18).
all
is
successfully
modeling
the
FG
building
parameters.
In
solving
a
Lasso
regression
problem
the
regularization
r defined hyperparameter. Using the above prior
final i estimate is computed using the p
following equation:
value,
while
other parameters
were set
to
their average value.
optimization
methods
to
solve
for
.
However,
the
E+
i
3.
Sampled
P(θ
)
using
its
CDF
and
a
uniform
distribution
on
the
tions
sampled
without
replacement,
and
evaluated
the
model
few
samples.
This
allowed
the
authors
to
use
conventional
tions
for
all
variables.
Using
the FG data set,of
we Energy
built a GGM
using 250
simula- No. DE-AC05-00OR22725. Our
i multiple
Byfolcomputing
toanalyzing
the ✓i[16].
’s
additional
derivations,
Fan Lito
[16]be
derived
the
U.S.
Department
under
Contract
numerical
methods
applicable.
Given,
EqMLE
17’ssamples
com-Maccording
fact
Figures
1(a)
indicates
that
the
GGM
isolates
hyperparameter
set
to
The
original
authors
i
This
results
in
two
simulations
per
building
parameter.
The
interval
[0,1]
toset
compute
datadefined
used
in this
work
several
million
data
vec1
using
150
test
simulations
simulations.
In
addition,
we
built
1 X
in Eq.
arecontains
able
to
estimate
optimization
methods
to
solve
for
.
However,
the
E+
ximum a posteriori (MAP) distributions fordistribution
all
115, we
i
tions
sampled
without
replacement,
and
evaluated
the
model
ˆj Pthe
to
use
sampling
=formulations
(✓j )to
(19) microarrays,
building
parameter’s
means
very
well.
While
the
uni1
1plex nature and Li’s [16] recommendation
i
derived
these
work
with
which
MO2
data
set
contains
minimum
and
maximum
work
has
been
enabled
and
by NSF funded RDAV Center of the
:
E[ i |✓i ,tors,
using1 mostly
weighted
sampling.
Inconventional
order to use optimization
invalidates
apM set
i , D]which
6. DISCUSSION
a GGM
using
the entire
MO1
data
set. pairwise
The
resulting
modelInsupported
1
data
data
vecj=1 used in this work contains several million
using
150
test
simulations
simulations.
addition,
we
built
to approximate i , we elected
to sampled,
use
E[ wei sample
|✓
, D]
as10,000
2the
i , contain
i Eq.
form
distribution
matches
mean
and
variances
(Figure
weight
15
using
its
CDF
andgrafting
a variables,
typically
to
12,000
but
have
very
combinations
across
theentire
sameMO1
building
parameter
set.
proaches.
Rather
than
using
the
fast
algorithm
While
the
parameters
in
FG and
MO2
have
well
defined
is
evaluated
using
300
sampled
MO2
simulations.
tors,
which
mostly
invalidates
conventional
optimization
ap1
1
University
of
Tennessee,
Knoxville
(NSF
grant no. ARRA-NSF-OCI-0906324
a
GGM
using
the
data
set.
The
resulting
model
D)our
/ estimates for
ˆ
unifrom
distribution
on
the
interval
[0,1].
This
means
our
,
which
is
the
MAP
estimate
under
a
where
M
is
the
total
number
of
samples
and
is
a
sample
1(b)),
it
appears
the
matching
is
only
possible
because
the
i
2
means,
they
have
instances
where
they
significantly
devii
samples.
This
thethan
authors
to
use
conventional
usedfew
by [16],
we solve
theallowed
Lasso
regression
problems
using
PN
PN
p
Using
FG data
set,300
wesampled
built a MO2
GGM
using 250 simulaAnalyzing Figure
1the
illustrates
that
the
building
paramproaches.
Rather
using
the
fast
grafting
algorithm
D
2
is
evaluated
using
simulations.
computed
using
Eq.
18.
Afterward
a
few
iterations
between
(xni
ij xnj ) +
i
ij |
ate
from their have
estimated
means,distribution
which is not
well
implied
Gamma
distribution.
Based
on| Fan
Li’s
[16]
MAP distribun=1
j=i+1
j=i+1
and
NSF-OCI-1136246).
actual
building
parameters
a
large
range,
1methods to
ADMM
(Section
4.1).
optimization
solve
for
.
However,
the
E+
eter
estimates
using
the
GGM
are
mostly
better than
thethe building
)
i
tions
sampled
without
replacement,
and
evaluated
the model
2
estimating
s and used
s, we
use [16],
the final
estimates
to comby
we
solve the
Lasso
regression
problems
using
Analyzing
Figure
1
illustrates
that
paramA
gradient
based
constrained
set
optimization
method.
by
results
(Figure
1(a)
and
2).
Figure
3(a)
illustrates
that
i
tion, we derived the following maximum
likelihood
estimate
which
mostly
centers
at
0.5,
the
uniform
distribution’s
exAfter
obtaining
’s
MAP
estimate,
it
can
be
used
to
pute
the
GGM’s
precision
matrix
(Eq.
7).
generated
bytest
randomly
guessing
using
a [0,are
1] In
randata set used
ini this(Section
work contains
several million data estimates
vec(16)
using 150
simulations
simulations.
addition,
wethan
builtthe
ADMM
4.1).
eter
estimates
using
the
GGM
mostly
better
1
1
(MLE) equation for i :
maximize
P ( i mostly
|✓i After
, simulations
with
respect
to
However,
pected
value.
i , D)
i . estimate,
distribution.
The
overall
GGM’sMO1
absolute
error
rate resulting
tors,
which
invalidates
conventional
optimization
apobtaining
’s
MAP
it brute
candom
be force
used
to
1.  Fine Grain (FG): contains
building
built
from
small
a
GGM
using
the
entire
data
set.
The
model
i
estimates
generated
by
randomly
guessing
using a [0,
1] ran1
5.
RESULTS
2 the Bayesian
1 the hyperparameter
P ( proaches.
ismaximize
dependent
upon
Similar
to
the
FGrespect
data✓algorithm
GGM
1
1
1
iset results,
is. statistically
better
with
95% confidence
than
the GGM’s
uniform
i |✓i , i , D) Rather
P
(
|✓
,
,
D)
with
to
However,
than
using
the
fast
grafting
i , D) / P (D| i , i , ✓i )P ( i | i , ✓i )P ( i |✓i )
i
i
i
dom
distribution.
The
overall
absolute error rate
is
evaluated
using
300
sampled
MO2
simulations.
i
In
orderLito [16]
assess
how well
the GGM
estimates
building
increments
across
14
building
parameters.
250
simulations
were
sampled
1
+
N
2i
+
|D|
and
Fan
noted
that
there
is
not
a
known
method
1
distribution’s
error
– 4.05±0.98
vs 5.07±1.01.
However,
1+ 1 + N 2i + |D|
presents
excellent
performance
onusing
the MO2 data
setisrate
(Figure
P
(
|✓
,
,
D)
is
dependent
upon
the
hyperparameter
✓
used
by
[16],
we
solve
the
Lasso
regression
problems
i
i
i
parameters,
we
experimented
with
three
data
sets
–
Fine
statistically
better
with
95%
confidence
than the
uniform
=
Analyzing
Figure
1
illustrates
that
the building
parami
P
P
P
i
mma(
,
N
D to analytically
N
1
1
remove
the
hyperparameter
via
integration.
2
2
onlythan
thethe
error
rates
for
variablesC.
1, 2,
3, 8, 10,
11,
12, 13, J. Nevins, G. Yao, and M. West. Sparse
2).
Bayesian
method
is
better
a uniform
distri(FG),
Model
Order
1 (MO1),
and The
Model
Order
2there
2
without
and
150
test
were
used
to
evaluate
✓i replacement,
+ ✓i + n=1 (xGrain
xnjsimulations
)4.1).
A.
Dobra,
Hans,
B.
Jones,
and
Fan
Li
[16]
noted
that
is
not
a
known
method
niADMM
ij
distribution’s
error
rate
–
4.05±0.98
vs 5.07±1.01.
However,
ij
(Section
j=i+1
j=i+1
P
P
eter
estimates
using
the
GGM
are
mostly
better
than
the
D
N
1
1
2
2
This
means
numerical
methods
are
required
to
approximate
(MO2).
The
first
data
set
contains
building
simulations
and
14
are
statistically
better
than
their
random
guessing
✓
+
✓
+
(x
x
)
ni
j
nj
bution
at
estimating
all
building
parameters.
The
Bayesian
ij
i
i
1
n=1
j=i+1
to(18)
analytically
remove
the hyperparameter
via integration.
onlygraphical
the error by
rates
for variables
1, 2, using
3, 8, 10,
11,
12,
13,
After
obtaining
it
can
be
used
to
)
GGM
model.
built
from
small
brute
force
increments
acrossestimate,
14
building
i ’s MAP
estimates
generated
randomly
guessing
a
[0,
1]
ranmodels
for
exploring
gene
expression
data. Journal of
the
integral
over
the
hyperparameter.
There
are
many
nucounterpart.
The
other
variables
are
not
statistically
di↵er2
1
1
GGM
has
better
overall
error
—
13.01±0.85
vs
41.6936±2.13.
1
This
means
numerical
methods
are
required
to
approximate
The MO1
data |✓
set ,contains
299with
building
simuGiven a fixed ✓i , we can estimate
a parameters.
sample
by
comand 14 are statistically
better
than absolute
their random
guessing
(17) merical
maximize
P
(
,
D)
respect
to
.
However,
i
i
i
i
dom
distribution.
The
overall
GGM’s
error
rate
methods
for
computing
approximates
to
definite
ini
ent.
2.
Model
Order
1
(MO1):
contains
299
building
simulations
and
~150
building
Multivariate
Analysis,
90(1):196–212,
2004.
lations
and
⇠150
building
parameters.
The
building
paramIn
addition,
the
model
produces
statistically
better
predic1
the
integral
over theupon
hyperparameter.
There are ✓many nucounterpart.
The
other
variables
are
not
statistically
di↵erputing
the
maximum
likelihood
estimate
(MLE)
(Eq.
18).
P
(
|✓
,
,
D)
is
dependent
the
hyperparameter
such
and
Simpson’s Rule.
ng P ( i | i , D) with respect to i is equivalent to tegrals,
ias iTrapezoidal
i
statistically
with
95% confidence
eters were
set to their method
minimum or
maximum
While
theis other
variablesbetter
are not
statistically
di↵erent,than the uniform
i independently
tions
for
all
variables.
merical
methods
for
computing
approximates
to
definite
inparameters.
The
building
parameters
were
independently
to their
min or a clearF.
ent.
By regression
computing
multiple
samples
according
to
the
✓iover
’snoted
Lasso
with the MLE
regularization
Li, Y.
Yang,
E.
Xing.
From
lasso
regression to feature vector machine.
value,
other
were
tounbounded
theirthere
average value.
However,
the
integral
✓isetisthat
beandwhile
Fan
Liparameters
[16]
isfrom
notabove,
a set
known
method
pproblem
there is notdistribution’s
indication
that
the–and
GGM
model
fails
to
error
rate
4.05±0.98
vs
5.07±1.01.
However,
meter
set to
original
authors
tegrals,
such
as Trapezoidal
i [16]. The
This able
results in
simulations
per
building
parameter. The method and Simpson’s Rule.
While
theindicates
other variables
are not
statistically di↵erent,
distribution
defined
in
Eq.
15, other
wecause
are
totwoestimate
✓
’s
values
exist
in
the
interval
[0,1),
which
means
i
represent
these
variables.
This
that
the
GGM
overmax
value,
while
parameters
were
set
to
their
average
value.
This
to
analytically
remove
the
hyperparameter
via
integration.
Advances
Neural1,that
Information
Systems, 18:779, 2006.
only the
error
rates
forinvariables
2, the
3, 8,
10, model
11,Processing
12,fails
13,to
ese formulations
to work with microarrays, which
MO2 data set contains
pairwise the
minimum
and maximum
1
However,
integral
over
✓
is
unbounded
from
above,
bei
there
is
not
a
clear
indication
GGM
E[
|✓
,
,
D]
using
weighted
sampling.
In
order
to
use
i toi 12,000 variables, but have very
Eq.
17 must
be transformed
to
a bounded
integral for
the
6.
DISCUSSION
containi 10,000
all is successfully
modeling
the FG building
parameters.
Inrandom guessing
combinations
across
the same building
parameter
set. required
This
means
numerical
methods
are
to
approximate
and
14
are
statistically
better
than
their
results
in
two
simulations
per
building
parameter.
The
entire
MO1
data
set
cause
✓
’s
values
exist
in
the
interval
[0,1),
which
means
i applicable.
represent
these variables.
This
indicates
that the GGM overes.
This allowed
the authors
use conventional
Usingits
themethods
FG dataand
set,
weabe
built
a GGM using Given,
250 simulaweight
sampled,
we tosample
Eq. 15 numerical
using
CDF
to
Eq
17’s
comfact
analyzing
Figures
1(a)
indicates
that
the
GGM
isolates
While
the
parameters
in
FG
and
MO2
have
well
defined
the
integral
over
the
hyperparameter.
There
are many
nu- for the
counterpart.
The othermodeling
variablesthe
areFG
notbuilding
statistically
di↵er-In
on methods to solve for i . However, the E+
Eq.
17
must
be
transformed
to
a
bounded
integral
tions
sampled
without
replacement,
and
evaluated
the
model
all
is
successfully
parameters.
plex
nature
Li’sour
[16] recommendation
to
use instances
sampling where
unifrom distribution
ontothe
interval
[0,1].
This and
means
used
build
GGM
model.
the
building
parameter’s
means very well. While the unised in this workwas
contains
several million
data vecmeans,
they
have
they
significantly
deviusing
150 test methods
simulations
simulations.
In
addition,
we
built
merical
for
computing
approximates
to
definite
inent.
1
1
numerical
methods
to E[
be applicable.
Eq
17’s
comfact analyzing
Figures
1(a)
indicates
that the GGM isolates
,MO1
wedata
elected
toresulting
use
|✓i , i , D] Given,
as
form
distribution
matches
the
mean
and
variances
(Figure
h mostly invalidates conventional optimization ap- to aapproximate
GGM
using
the
entire
set.
The
model
i
i
ate
from
their
estimated
means,
which
is not While
well the
implied
3.  Model
Order
2 (MO2):
contains
pairwise
min
and
max
combinations
across
tegrals,
such
as1 , nature
Trapezoidal
method
and
Simpson’s
Rule.
2
the
otherisparameter’s
variables
are
not very
statistically
di↵erent,
plex
and
Li’s
[16]
recommendation
to
use
sampling
building
means
well. While
the uniRather
than
using
the
fast
grafting
algorithm
is
evaluated
using
300
sampled
MO2
simulations.
our
estimates
for
which
is
the
MAP
estimate
under
a
1(b)),
it
appears
the
matching
only
possible
because
the
2
i
1
1
based
constrained
set parameter
optimization
method.
bytheresults
(Figure
1(a)
andabove,
2).were
Figure
3(a)
illustrates
that
16], A
we gradient
solve the
Lasso
regression
problems using
However,
theto
integral
over
✓building
isMO2
from
Analyzing
Figure
1300
illustrates
that
paramapproximate
, unbounded
we
elected
to
use
E[
|✓beD] as
iiLi’s
there
is
not
a clear
indication
that
GGM
model fails
to
i , used
i , building
form
distribution
matches
the the
mean
and variances
(Figure
the
same
building
set.
sampled
simulations
Gamma
distribution.
Based
on
Fan
[16]
MAP
distribui actual
parameters
have
a
large
distribution
range,
1 than the
Section 4.1).
eter
estimates
using
the GGM
are mostly
better
cause
✓
’s
values
exist
in
the
interval
whichestimate
means
our
estimates
for
which [0,1),
is the
MAP
under
a centers
i
represent
these
variables.
indicates
that
the GGM
over-the
1(b)),
it appears
theThis
matching
is only
because
tion,
we
derived
the
following
maximum
likelihood
estimate
i a ,[0,
which
mostly
at 0.5,
the uniform
distribution’s
ex-possible
btaining i ’s MAP
estimate,
it
can
be
used
to
estimates
generated
by
randomly
guessing
using
1]
ranto
evaluate
the
GGM
model
built
from
the
entire
MO1
data
set.
1 distribution.
Gamma
on Fanintegral
Li’s [16]for
MAP
Eq.
17 must
beoverall
transformed
to Based
a bounded
thedistribuP ( i 1 |✓i , i , D) with respect to i . However, (MLE)
actual building
parameters
havebuilding
a large distribution
equation
for
all is successfully
modeling
the FG
parameters.range,
In
dom
distribution.
The
pected
value.
i : GGM’s absolute error rate
i , D) is dependent upon the hyperparameter ✓i
is statistically
better
with 95%
than
uniformGiven,
tion,
we confidence
derived
the the
following
maximum
estimate
which
centers
at
0.5, the that
uniform
numerical
methods
to be applicable.
Eq likelihood
17’s comSimilar
to
theanalyzing
FG
datamostly
set
results,
the
Bayesian
GGM
fact
Figures
1(a)
indicates
the distribution’s
GGM isolatesexLi [16] noted that there is not a known method
1
distribution’s error rate
–
4.05±0.98
vs
5.07±1.01.
However,
(MLE) equation for
:
Acknowledgments
Data Sets
References