Nicholas   Cross,   Nigel   Hambly,   Ross   Collins,
Eckhard   Sutorius,   Mike   Read   and   Rob   Blake.
njc@roe.ac.uk

y Must   be   able   to   select   useful   variables   and   non ‐ variables   from   catalogues   of   millions   (billions)   of   objects.
y Important   plots,   such   as   magnitude ‐ RMS   and   light   curves   must   be   easily   generated   using   simple   SQL   statements y Steps   that   many   (most)   users   will   take   should   be   done   by   archive   team.
y More   specialised   processing   left   to   individual   scientists.
y Processing   split   up   to   enable   parallelisation.
y Automation   allows   large   number   of   small   programmes   to   be   mass   produced   and   prevents   operators   from   forgetting   particular   stages.
Easy   to   add   in   new   steps   if   needed.
NGSS,   Edinburgh
07/06/2009 2

‐
y y y y y y y
Periodic   variables   as   distance   indicators   to   map   out   3 ‐ D   structure   of   bulge   and   Magellanic   Clouds:   VVV   &   VMC y VVV  ‐ 1   billion   sources,   100   epochs
Supernovae :   VIDEO
High   proper   motion   stars :   DXS   (Not   main   science   goal,   but   secondary   use   of   data).
Transits   of   planets   around   M ‐ dwarfs :   Campaign   2   data
YSOs (Alves   de   Oliveira   &   Casali   2008):   non ‐ survey   data
Better   QC and   flagging .
All   deep   data   (all   data   if   mistakes   are   found).
Calibration,   new   faint   near ‐ IR   standards :   WFCAM   /   VISTA   standard   star   data
07/06/2009 NGSS,   Edinburgh 3

y Rapid   follow   up   required?
– VOEvent y y y y y
Time   from   observation   to   archiving:   6+   weeks   for   VDFS
Time   needed   for   calibration:   1   month
Release   schedule:   every   6   months y Difference   imaging   required   for   very   quick   follow   up:   this   must   be   done   at   the   telescope.
Transients
Low
– many amplitude
σ :   calibration variables calibrated   photometry.
need
less good
important.
QC   and   well   y Science   archives transients.
ideal   for   variables,   too   late   for   many
07/06/2009 NGSS,   Edinburgh 4

y y y y y y
Unique   master   source   list   to   compare   each   epoch   to   and   to   match   to   external   surveys
Best   match   between   each   source   and   each   observation   so   that   light/motion   curves   can   be   produced
Merging   observations   in   different   filters   where   the   interval   is   short   compared   to   the   variability   under   study
(correlated   filters)
Statistical   analysis   of   individual   epoch   observations   of   each   source
Classification   of   each   source   based   on   its   statistical   properties   and   the   noise   properties   of   the   observations
List   of   bright   unmatched   objects   for   fast ‐ moving   objects,   transients   etc.
07/06/2009 NGSS,   Edinburgh 5

y VDFS:   Reseamed   merged   filter   catalogues   from   deep   stacks.
y Advantages: y y y y y y
Uses   existing   software
Useful   for   deep   programmes
Simple   to   understand
Matches   to   objects   that   are   too   faint   to   be   detected   in   one   observation   (can   be   a   disadvantage   in   time)
Disadvantages: y y
Slow   – need   to   wait   until   deep   stacks   are   produced
Misses   objects   that   are   only   visible   in   a   few   frames
Alternative:   PanSTARRS:   2+   5 σ detections
07/06/2009 NGSS,   Edinburgh 6

y Possible   solutions: y 1)   Nearest   match   y 2)   List ‐ driven   photometry y 3)   Magnitude   and   position   selection y 4)   Motion   based   match   (e.g.
linear   PM   +   parallax   or
MOPS   which   checks   if   SSOs have   orbit   that   follows
Newton’s   laws).
y Missing   detections   – flagging    (time   consuming)   y VDFS   uses   1),   but   as   a   first   step   to   eventually   using   4).
07/06/2009 NGSS,   Edinburgh 7

Efficient   searches   to   see   if   missing   objects   are   on   the   detectors   or   within   dither   offset.
This   is   one   of   the   rate   determining   steps.
07/06/2009 NGSS,   Edinburgh 8

• Unique   catalogue   from   deep   frames   ( Source )   is   linked   to   individual   epoch   detections   ( Detection )    through   BestMatch .
• Statistics   of   multi ‐ epoch   data   stored   in   Variability .
• Information   from   framesets   (magnitude   limits   and   fits   to   the   rms are   in   VarFrameSetInfo .
• If   a   photometric   recalibration   occurs,   the
BestMatch table   is   not   changed.
07/06/2009 NGSS,   Edinburgh 9

y Many   surveys   have   different   filters   taken   close   together:   SDSS,   VDFS   calibration   fields y Merging   filters   makes   it   easier   for   users   to   see   colour   changes   over   time   which   can   distinguish   different   types   of   variables y In   VDFS   we   merge   the   filters   together   using   same   routines   as   we   do   for   merging   filters   for   main   object   table,   but   interval   to   merge   over   must   be   supplied   in   survey   requirements
07/06/2009 NGSS,   Edinburgh 10

y Two   additional   tables y SynopticMergeLog y SynopticSource y SynopticSource has   similar   attributes   as
Source ,   but   is   linked   like   Detection y SynopticMergeLog like
MergeLog ,   but   with   a   time   attribute.
07/06/2009 NGSS,   Edinburgh 11

• Two   variables
• Two   standard   stars:
¾ Ser ‐ EC   68
¾ Ser ‐ EC   84
07/06/2009 NGSS,   Edinburgh 12

y Fit   to   motion   – PM,   parallax,   SSO   orbital   parameters y Number   of   good,   flagged,   missing   detections y Cadence:   analysis   of   observation   period y Photometric   statistics:   mean,   rms,   median,   skewness,
Welch ‐ Stetson.
y Fourier   analysis   y Number   of   outliers y Many   light ‐ curves   are   sparsely   sampled.
07/06/2009 NGSS,   Edinburgh 13

Light   curves   of   two   non ‐ variables.
Variations   are
0.2
within   errors.
Increase   in   brightness   over   700   days   in   K.
Seems   to   rise   and   fall   in   J.
07/06/2009 NGSS,   Edinburgh 14

May   be   eclipsing   binary?
Cannot   get   well   defined   period   from   data.
Many   objects   have   sparsely,   and   unevenly   distributed   data   points.
Classification   must   be   robust   to   this.
07/06/2009 NGSS,   Edinburgh 15

y Objective   is   to   separate   as   many   different   types   of   objects.
y Need   accurate   noise   model   to   separate   low   amplitude   variables   from   noise.
y Use   all   statistical   attributes   – photometric,   astrometric and   time   series.
y VDFS   uses   ratio   of   intrinsic   to   expected   rms weighted   by   number   of   observations   in   each   filter y Use   all   other   data   (multi ‐ wavelength),   point/extended   source   separation   (Neighbour   tables).
07/06/2009 NGSS,   Edinburgh 16

NGSS,   Edinburgh y y
Empirical   fit
Strateva function
= a
+ b 10
0 .
4 m + c 10
0 .
8 m y y y
Fits   faint   end   well   but   underestimates   noise   at   bright   end   (saturation)
Assumes   all   epochs   have   same   noise   properties
Physical   model   better,   but   needs   more   work.
17 07/06/2009

y Variable:   At   least   5   good   detections   in   one   filter   and   weighted   ratio   of   intrinsic   rms to   noise   is   >=3.
y DXS:   0.6%   of   stars   classified   as   variable   (K<15   mag,
Δ K>0.015
mag.
2.3%   of   galaxies:   10 7 – 10 4 .
y More   sophisticated   analysis   will   use   skew,   W ‐ S,
Fourier   analysis,   motion   etc.
from   multi ‐ epoch   analysis y Classification   could   also   use   other   image   parameters   such   as   star ‐ galaxy   separation,   colour y Use   external   data   via   neighbour   tables   or   VO?
This   is   more   difficult   to   add   into   pipeline.
07/06/2009 NGSS,   Edinburgh 18

Astrometry   Mag ‐ Rms
Intrinsic   Rms vs skewness
WSA   gives   additional   useful   data   such   as   star ‐ galaxy   separation   and   links   to   external   surveys   through   neighbour   tables.
07/06/2009 NGSS,   Edinburgh
J ‐ K   vs K   colour   magnitude
19

Some   objects   show   astrometric variation   (PM),   BUT   difficult   to   search   on:   WILL   IMPROVE   IN   LATER   VERSIONS
07/06/2009 NGSS,   Edinburgh 20

y Use   bright   stars   to   recalibrate   each   intermediate   stack   detector   by   comparison   to   the   deep   stack   in   that   filter.
y If   | Δ ZP|>0.05
mag,   deprecate   detector   frame y <| Δ ZP|>~0.005
mag for
DXS   frames
07/06/2009 NGSS,   Edinburgh
Frame   that   had   slipped   through   since   EDR
Histogram   of   ∆ ZP
21

07/06/2009 NGSS,   Edinburgh
An   object   marked   as   variable
Two   overlapping   frame   sets:
∆ m~0.035
mag between    frames   on   each   side   of   overlap.
Some   instrument   effects   left   in.
More   calibration/QC   required.
22

y Small   programmes   automated   from   start   to   finish: y Requirements   derived   from   metadata y Pipeline   applies   QC   and   uses   requirements   to   drive   processing   up   to   synoptic   tables   (if   required) y Main   survey   requirements   and   QC   are   partly   specified   by   survey   science   team y Pipeline   uses   requirements   to   drive   processing   up   to   synoptic   tables.
y If   pipeline   crashes,   it   can   work   out   what   has   been   done   – consistency   checks  ‐ and   skips   steps   that   have   already   been   accomplished.
07/06/2009 NGSS,   Edinburgh 23

y List ‐ driven   photometry   added   in   – better   at   faint   end,   helps   sort   out   deblending y More   cadence   statistics y More   QC   and   consistency   tests.
y Additional   classification y Moving   objects y Difference   imaging y Fourier   analysis y Orphan   table
07/06/2009 NGSS,   Edinburgh 24

02/12/2008 St   Andrews   Seminar   Talk 25

y y
Searches   for   variability   possible   in   WSA   from
UKIDSS ‐ DR5   (April   2009+).
Non ‐ surveys   (some   released   already)   and   VSA   to   follow.
1 st version   of   synoptic   pipeline:   future   versions   will   improve   quality   and   quantity   of   attributes.
y y
Fitting   astrometric data   for   PM
Improved   noise   model y Improved   classification   (use   more   parameters) y y
Automated   pipeline   – able   to   do   full   pipeline   on   NS.
VSA:   massive   amounts   of   data   to   trawl   through   – need   efficient   search   capability.
E.g.
VVV y See   Cross   et   al.
2009   (astro ‐ ph:   0905.3073)
07/06/2009 NGSS,   Edinburgh 26