Nicholas
Cross,
Nigel
Hambly,
Ross
Collins,
Eckhard
Sutorius,
Mike
Read
and
Rob
Blake.
njc@roe.ac.uk
WFCAM
NIR
surveys
VISTA
• 
4
2k
x
2k
detectors
• 
0.21
sq.
deg
field
of
view
• 
0.4”
pixels
• 
Mauna
Kea
02/12/2008
St
Andrews
Seminar
Talk
•  16
2k
x
2k
detectors
•  0.6
sq.
deg
field
of
view
•  0.34”
pixels
•  Cerro
Paranal
2
VISTA
Data
Flow
System
  Processing
and
archiving
of
all
WFCAM
and
most
VIRCAM
data
  WFCAM
pathfinder
for
VISTA
  Observation
block
processing
by
CASU
in
Cambridge
+
calibration
  Later
processing
and
archiving
by
WFAU
in
Edinburgh.
02/12/2008
St
Andrews
Seminar
Talk
3
VDFS
mul:‐epoch
science
  Periodic
variables
as
distance
indicators
to
map
out
3‐D
structure
of
bulge
and
Magellanic
Clouds:
VVV
&
VMC
  Supernovae:
VIDEO
  High
proper
motion
stars:
UKIDSS‐DXS
(Not
main
science
goal,
but
secondary
use
of
data).
  Transits
of
planets
around
M‐dwarfs:
WFCAM
Campaign
data
  YSOs
(Alves
de
Oliveira
&
Casali
2008):
PI
data
  Calibration,
new
faint
near‐IR
standards:
WFCAM
/
VISTA
standard
star
data
  Separating
L/T
dwarfs
from
QSOs
using
proper
motion
(UKIDSS
‐
LAS,
GCS)
  All
scientists
from
across
ESO
countries
have
access:
a
user
community
whose
science
goals
go
well
beyond
survey
science
teams
20/01/2010
NGSS,
QUB
4
VISTA
Surveys.
VVV:
500
sq.
deg,
100
epochs;
VMC:
180
sq.
deg,
12
epochs;
VIDEO
10
sq.
deg,
10s
epochs;
Ultra‐VISTA
1
sq.
deg,
100s
of
epochs
Transients
vs
Variables
  Transients:
Rapid
follow
up
required?
  Time
from
observation
to
archiving:
6+
weeks
for
VDFS
  Time
needed
for
calibration:
1
month
  Release
schedule:
every
6
months
  Difference
imaging
required
for
very
quick
follow
up:
this
must
be
done
at
the
telescope.
  Transients
–
many
σ:
calibration
less
important.
  Low
amplitude
variables
need
good
QC
and
well
calibrated
photometry.
  Science
archives
ideal
for
variables,
too
late
for
many
transients.
20/01/2010
NGSS,
QUB
6
Archiving
Mul:‐epoch
data
  Must
be
able
to
select
useful
variables
and
non‐variables
from
catalogues
of
millions
(billions)
of
objects.
  Important
plots,
such
as
magnitude‐RMS
and
light
curves
must
be
easily
generated
using
simple
SQL
statements
  Steps
that
many
(most)
users
will
take
should
be
done
by
archive
team.
  More
specialised
processing
left
to
individual
scientists.
  Processing
split
up
to
enable
parallelisation.
  Automation
allows
large
number
of
small
programmes
to
be
mass
produced
and
prevents
operators
from
forgetting
particular
stages.
Easy
to
add
in
new
steps
if
needed.
  VISTA
surveys,
Range
of
data,
very
heterogeneous
and
different
science
goals
20/01/2010
NGSS,
QUB
7
Time
Series
Analysis
  Users
want
to
know:
  What
timescales
can
we
analyse
the
data
on?
  Is
source
measurably
variable?
  Is
the
source
measurably
moving?
  What
type
of
object
is
this
source?





20/01/2010
Higher
order
statistics
(skew)
(Cepheids,
eclip
bin)
Min,
max
vs
median
(Eclip
bin
vs.
flare)
Correlations
between
filters
(IWS)
(Anti
–
correlation)
Fourier
analysis
Additional
data
(colours,
star‐galaxy,
environment)
–
archive/
VO
NGSS,
QUB
8
Elements
of
a
Synop:c
Survey
  Unique
master
source
list
to
compare
each
epoch
to
and
to
match
to
external
surveys
  Best
match
between
each
source
and
each
observation
so
that
light/motion
curves
can
be
produced
(linear
motion)
  Merging
observations
in
different
filters
where
the
interval
is
short
compared
to
repeat
observations
of
the
same
filter
(correlated
filters)
  Statistical
analysis
of
individual
epoch
observations
of
each
source
  Classification
of
each
source
based
on
its
statistical
properties
and
the
noise
properties
of
the
observations
  List
of
bright
unmatched
objects
for
fast‐moving
objects,
transients
etc.
20/01/2010
NGSS,
QUB
9
Synop:c
table
design
• 
Unique
catalogue
from
deep
frames
(Source)
is
linked
to
individual
observations
(Detection)
through
BestMatch.
• 
Statistics
of
multi‐epoch
data
stored
in
Variability.
• 
Information
from
framesets
(magnitude
limits
and
fits
to
the
rms
are
in
VarFrameSetInfo.
• 
If
a
photometric
recalibration
occurs,
the
BestMatch
table
is
not
changed.
02/12/2008
St
Andrews
Seminar
Talk
10
Correlated
Filters
  Two
additional
tables
  SynopticMergeLog
  SynopticSource
  SynopticSource
has
similar
attributes
as
Source,
but
is
linked
like
Detection
  SynopticMergeLog
like
MergeLog,
but
with
a
time
attribute.
02/12/2008
St
Andrews
Seminar
Talk
11
Important
features
  Quality
control
(image
and
catalogue
level)
  Calibration
(internal
consistency
and
external)
  Parameters
to
allow
users
to
quickly
decide
on
the
usefulness
of
data
(NgoodObs,
cadence
etc)
  Specified
model
to
link
individual
observations
to
overall
source
list.
  Well
specified
separation
of
variable
/
non‐variable
  Classification
into
different
types
of
variable
  Archive
uses
other
info
from
same
surveys
–
star/
galaxy
classification,
deep
image
parameters,
environment
parameters
etc.
  Link
to
external
data
for
additional
science
(Neighbour
tables,
VO
etc)
20/01/2010
NGSS,
QUB
12
Recalibra:on
and
QC
  Use
bright
stars
to
recalibrate
each
epoch
observation
by
comparison
to
the
deep
stack
in
that
filter.
  If
|ΔZP|>0.05
mag,
deprecate
detector
frame
  <|ΔZP|>~0.005
mag
for
DXS
frames
Frame
that
had
slipped
through
since
EDR
Histogram
of
ΔZP
20/01/2010
NGSS,
QUB
13
Noise
model
  Empirical
fit
  Strateva
function
  Fits
faint
end
well
but
underestimates
noise
at
bright
end
(saturation)
  Assumes
all
epochs
have
same
noise
properties
  Physical
model
better,
but
needs
more
work.
  DXS:
rms
~
0.004
mag
20/01/2010
NGSS,
QUB
14
Classifying
Variables
Light
curves
of
two
non‐
Light
curve
of
variable
galaxy.
0.2
variables.
Variations
are
mag
linear
within
errors.
Increase
in
brightness
over
700
days
in
K.
Seems
to
rise
and
fall
in
J.
20/01/2010
NGSS,
QUB
15
Correlated
light
curves
• 
Two
variables
• 
Two
standard
stars:
 Ser‐EC
68
 Ser‐EC
84
20/01/2010
NGSS,
QUB
16
Classifica:on
Astrometry
Mag‐Rms
Intrinsic
Rms
vs
skewness
WSA
gives
additional
useful
data
such
as
star‐galaxy
separation
and
links
to
external
surveys
through
neighbour
tables.
J‐K
vs
K
colour
magnitude
20/01/2010
NGSS,
QUB
17
VVV:
very
dense
•  Large
catalogues
1011
•
•
•
•
•
detections,
109
sources
Deblending
Extinction
500
sq.
deg.
Cadence
~1
day
(100
epochs
over
a
few
months)
Periodic
variables
(Cepheids,
RR
Lyrae)
20/01/2010
NGSS,
QUB
18
Example
WFCAM
Standards
20/01/2010
NGSS,
QUB
19
Issues
for
mee:ng
  ‐
what
is
the
scientific
and
technical
expertise
you
have
developed
  Designing
a
dynamical
relational
model
for
a
wide
range
of
multi‐epoch
data,
that
includes
an
empirical
noise
model
and
a
wide
range
of
useful
parameters
to
select
on.
  Building
standard
methods
directly
into
archive
curation
scripts,
so
users
can
search
on
variability.
  Automation
of
archive
curation
scripts
to
allow
processing
of
wide
range
of
small,
medium
and
large
programmes
with
different
timescales,
filters,
and
number
of
epochs.
  ‐
what
are
the
key
computational
challenges
in
your
time
domain
surveys,
both
current
and
future
  Fast
matching
of
observations
where
there
is
no
detection.
  Processing
VVV
in
sensible
timescale.
1011
detections
(May
have
to
break
detection
table
into
parts).
  Classification
of
different
types
of
variable.
Going
from
light‐curve
analysis
+
colours
etc
to
physical
classification.
  See
Cross
et
al.
2009,
MNRAS,
399,
1730
for
details
of
work
so
far.
20/01/2010
NGSS,
QUB
20
Future
items
  More
cadence
statistics
  More
QC
and
consistency
tests.
  Additional
classification:
different
types
of
variable
  Moving
objects
  Difference
imaging
  Fourier
analysis
  Orphan
table
20/01/2010
NGSS,
QUB
21