Pan-STARRS1
Transient Database
K. Smith, D. R. Young, S. Valenti, S. Smartt
Wednesday, 27 October 2010
PanSTARRS1 Overview
• 1.8m telescope
• 1.4 Gigapixel Camera
• 7 square deg FOV
• Image Processing Pipeline (IPP)
• Few Tb of data every night
Wednesday, 27 October 2010
Survey Modes : Medium Deep
Wednesday, 27 October 2010
Survey Modes : Medium Deep
• 10 x 7 squ. deg. fields
• Observed in g,r,i,z,Y every 4
nights
• 113 (g,r) or 240 (i,z,Y) sec
exposures
• Well known fields
• 8 exposures/field daily
• 4 or 5 fields observed daily
Wednesday, 27 October 2010
Survey Modes : Medium Deep
QUB MD Transient Processing
Stack–Stack Differencing
• Nightly stacks of 8MD exposures per MD field produced by IPP
• Differenced with a Reference Stack by IPP
•(may be another nightly stack or deep stack)
• Download & process 19MB/day StackStack Difference catalogues
• (33, 40, now 55 columns of info per FITS table + header)
• 80,000 difference objects/day
• MD Transient Database grows by 27MB/day
• Download & store 200MB/day “postage stamp” images
• Daily download of IPP (GPC1) database (4GB compressed)
Wednesday, 27 October 2010
Current Medium Deep Coverage
Wednesday, 27 October 2010
Processing PS1
Data
Wednesday, 27 October 2010
Processing PS1
Data
Wednesday, 27 October 2010
Download Catalogue data
Processing PS1
Data
Wednesday, 27 October 2010
Download Catalogue data
Stack-Stack diff catalogues
Processing PS1
Data
Wednesday, 27 October 2010
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Stack-Stack diff catalogues
Processing PS1
Data
Wednesday, 27 October 2010
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Human eyeballing (+ Galaxy Zoo)
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Human eyeballing (+ Galaxy Zoo)
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Human eyeballing (+ Galaxy Zoo)
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Human eyeballing (+ Galaxy Zoo)
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Human eyeballing (+ Galaxy Zoo)
Promote and Assign Object Name
or Discard Object
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Human eyeballing (+ Galaxy Zoo)
Promote and Assign Object Name
or Discard Object
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Human eyeballing (+ Galaxy Zoo)
Promote and Assign Object Name
or Discard Object
Crossmatch Promoted Objects
(e.g. with CfA)
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Human eyeballing (+ Galaxy Zoo)
Promote and Assign Object Name
or Discard Object
Crossmatch Promoted Objects
(e.g. with CfA)
Wednesday, 27 October 2010
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Automated
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Human eyeballing (+ Galaxy Zoo)
Promote and Assign Object Name
or Discard Object
Automated
Wednesday, 27 October 2010
Crossmatch Promoted Objects
(e.g. with CfA)
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Processing PS1
Data
Download Catalogue data
Pre-Ingest Cuts, Run Crossmatch &
Classifier Algorithm, Ingest Data
Post-Ingest Quality Cuts &
Post Quality Candidates to Eyeball
Automated
Generate Lightcurve & Scatter Plots
for Quality Candidates
Request Images for Quality
Candidates
Poll for & download images
Human eyeballing (+ Galaxy Zoo)
Manual
Promote and Assign Object Name
or Discard Object
Automated
Wednesday, 27 October 2010
Crossmatch Promoted Objects
(e.g. with CfA)
Stack-Stack diff catalogues
Supernova, AGN, NT,
Variable Star, Orphan
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Software Components
Wednesday, 27 October 2010
Software Components
CentOS
5
(RHEL
5)
Wednesday, 27 October 2010
Software Components
MySQL
Database
CentOS
5
(RHEL
5)
Wednesday, 27 October 2010
Software Components
HTTP
PS1
Difference
Catalogues
(FITS)
PS1 Image Processing
Pipeline
MySQL
Database
CentOS
5
(RHEL
5)
Wednesday, 27 October 2010
Software Components
PS1 Image Processing
Pipeline
HTTP
CCFits
CFITSIO
PS1
Difference
Catalogues
(FITS)
MySQL
Database
CentOS
5
(RHEL
5)
Wednesday, 27 October 2010
Software Components
PS1 Image Processing
Pipeline
HTTP
CCFits
HTM
MySQL++
CFITSIO
PS1
Difference
Catalogues
(FITS)
MySQL
API
MySQL
Database
CentOS
5
(RHEL
5)
Wednesday, 27 October 2010
Software Components
PS1 Image Processing
Pipeline
HTTP
Command
Line/CRON
PERL
Wrapper
/
File
SpliLer
C++
Ingester
/
Classifier
CCFits
HTM
MySQL++
CFITSIO
PS1
Difference
Catalogues
(FITS)
MySQL
API
MySQL
Database
CentOS
5
(RHEL
5)
Wednesday, 27 October 2010
Software Components
PS1 Image Processing
Pipeline
HTTP
Command
Line/CRON
PERL
Wrapper
/
File
SpliLer
Python
PS1
Image
Request/Fetch
C++
Ingester
/
Classifier
CCFits
HTM
MySQL++
MySQLDB
Python
API
MySQL
API
MySQL
Database
CFITSIO
PS1
Difference
Catalogues
(FITS)
CentOS
5
(RHEL
5)
Wednesday, 27 October 2010
Software Components
PS1 Image Processing
Pipeline
HTTP
PS1 Postage Stamp
Server
HTTP
Command
Line/CRON
PERL
Wrapper
/
File
SpliLer
Python
PS1
Image
Request/Fetch
C++
Ingester
/
Classifier
CCFits
HTM
MySQL++
MySQLDB
Python
API
MySQL
API
MySQL
Database
CFITSIO
PS1
Difference
Catalogues
(FITS)
CentOS
5
(RHEL
5)
Wednesday, 27 October 2010
PS1
Image
Stamps
Software Components
PS1 Image Processing
Pipeline
HTTP
PS1 Postage Stamp
Server
Command
Line/CRON
Web
Interface
PERL
Wrapper
/
File
SpliLer
Django
Framework
C++
Ingester
/
Classifier
Apache
Python
PS1
Image
mod_python Request/Fetch
CCFits
HTM
MySQL++
MySQLDB
Python
API
MySQL
API
MySQL
Database
CFITSIO
PS1
Difference
Catalogues
(FITS)
CentOS
5
(RHEL
5)
Wednesday, 27 October 2010
HTTP
PS1
Image
Stamps
Hardware
Transient Server & Local IPP
DELL PowerEdge 2950 III (64GB)
8 Core Xeon X5460 (3.16 GHz)
1 TB (RAID 5) Internal SAS Storage
Transient Server Database Logs
& Temporary Tables
Web & File Server
DELL PowerEdge 2950 III (2GB)
4 Core Xeon E5405 (2.0 GHz)
DELL PowerVault MD3000
5.3 TB (RAID 5) SAS Storage Array
Transient Server Database Tables
DELL PowerVault MD1000
11.2 TB (RAID 5) SATA Storage Array
Database Backup Staging
Image Stamps
Follow-up Data
Wednesday, 27 October 2010
DELL PowerVault MD1000
11.2 TB (RAID 5) SATA Storage Array
Backup Mirror
IPP Data Products (catalogue files)
External Catalogue Ingestion
Wednesday, 27 October 2010
External Catalogue Ingestion
8 Generic Catalogues
2 x 109 rows
0.5 TBytes (MyISAM)
Veron 12
2MASS XSC
SDSS Stars
2MASS PSC
Guide Star 2.3
NED
SDSS Photo Galaxies
SDSS Spectroscopic Galaxies
Wednesday, 27 October 2010
External Catalogue Ingestion
8 Generic Catalogues
> 20 Medium Deep Field Catalogues
109
2x
rows
0.5 TBytes (MyISAM)
> 40 views
105
3x
rows
0.1 TBytes (MyISAM)
MD01 NED
Veron 12
MD05 NED
MD01 Chiappetti 2005
2MASS XSC
MD05 Brunner 2008
MD06 NED
MD01 Pierre 2007
SDSS Stars
MD02 NED
2MASS PSC
Guide Star 2.3
NED
SDSS Photo Galaxies
MD07 NED
MD02 Giacconi 2002
MD07 Laird 2009
MD02 LeFevre 2004
MD07 Nandra 2005
MD02 Leher 2005
MD08 NED
MD02 Virani 2006
SDSS Spectroscopic Galaxies
MD08 Manners 2003
MD03 NED
MD09 NED
MD04 NED
MD04 Hasinger 2007
MD04 Trump 2007
Wednesday, 27 October 2010
MD10 NED
External Catalogue Ingestion
8 Generic Catalogues
> 20 Medium Deep Field Catalogues
109
2x
rows
0.5 TBytes (MyISAM)
> 40 views
105
3x
rows
0.1 TBytes (MyISAM)
MD01 NED
Veron 12
MD05 NED
MD01 Chiappetti 2005
2MASS XSC
MD05 Brunner 2008
MD06 NED
MD01 Pierre 2007
SDSS Stars
MD02 NED
2MASS PSC
Guide Star 2.3
NED
SDSS Photo Galaxies
MD07 NED
MD02 Giacconi 2002
MD07 Laird 2009
MD02 LeFevre 2004
MD07 Nandra 2005
MD02 Leher 2005
MD08 NED
MD02 Virani 2006
SDSS Spectroscopic Galaxies
MD08 Manners 2003
MD03 NED
MD09 NED
MD04 NED
MD10 NED
MD04 Hasinger 2007
MD04 Trump 2007
• Source data in various formats (CSV, FITS,VO Tables)
• Custom ingest scripts written (PERL, C++, Java)
• Policy adopted of NaN, ±∞, null values = NULL
• Unit sphere cartesian coords (cx, cy, cz) calculated on ingest for cone searches
• HTM IDs for each data point calculated on ingest
Wednesday, 27 October 2010
Local MySQL Databases
Crossmatch
Catalogues
IPP
GPC1
Ingest
Database
(views
onto
Crossmatch
Database)
Access
to
GPC1
Database
Django
Web
Database
(views
onto
Ingest
Database)
Wednesday, 27 October 2010
'(
))
%&
%&
'*%+,
'*%+,
2
!%&
3(-
'%&
-
'(
))
#
! *%+,
((
))
(#- (./++
##-
%&
Transient
Server
Database
Schema
'*%+,
'*%+,
#2
)
!('
#'(
)
'%&
'(
)
#
! *%+,
'(#
0('
'(#
)
'((1-2
##-
((
))
##-
%&
(',/,
!('./++
)
!%&
./++
(
"
(#- !
"
./++
(#- (./++
!%&
#
4
#
$
%&
!%&
!%&
!
"
Wednesday, 27 October 2010
!%&
$%
&*$+,
&*$+,
$%
&$%
"
''
())
""-
Transient
Server
Database
Schema
"2
)
"&'
)
&'
)
*$+,
""-
$%
&'
())
*$+,
&'
())
$%
&*$+,
&*$+,
'&
&$%
"
0'&
&''1-2
)
''
())
""-
&'"
&'"
)
'"- !
./++
'"-'./++
$%
$%
'&,/,
'&./++
$%
"
3
"
#
Wednesday, 27 October 2010
Crossmatching: JHU Hierarchical Triangular Mesh
Splits the Celestial Sphere into “Spherical Triangles” which are spatially numerically adjacent
e.g. Level 20 HTM id for RA, DEC = 243.334412243, 53.9783926374
N112212112030032100002
(1474714251366610)
circleRegion API returns a vector of HTM ID tuples
(high and low HTM ID)
Wednesday, 27 October 2010
HTM + Cartesian Cone Search
Wednesday, 27 October 2010
HTM + Cartesian Cone Search
Wednesday, 27 October 2010
HTM “circleRegion” + Cone Search
SELECT
*
FROM
catalogue
WHERE
(
htm_id BETWEEN id(0) AND id(1)
OR
htm_id BETWEEN id(2) AND id(3)
OR
.
.
OR
htm_id BETWEEN id(n-1) AND id(n)
)
AND (
cx * object x coordinate +
cy * object y coordinate +
cz * object z coordinate >= cos(radius in radians)
);
Wednesday, 27 October 2010
MySQL Database Organisation
MyISAM storage engine utilised
(Fast SELECT, fast INSERT, fast replication)
(*updates must be carefully managed - table-level locking)
RA/DEC and cx, cy, cz Coordinate columns indexed
HTM ID column indexed
HTM IDs spatially numerically adjacent, table ordered by HTM ID
(ALTER TABLE catalogue ORDER BY htmID)
Database tables stored on high speed, low latency disks (SAS)
(Indexes will eventually be stored on physically separate volumes)
Wednesday, 27 October 2010
Contextual Information: e.g. MD08
Wednesday, 27 October 2010
Contextual Information: e.g. MD08
Wednesday, 27 October 2010
Contextual Information: e.g. MD08
Wednesday, 27 October 2010
Contextual Information: e.g. MD08
Wednesday, 27 October 2010
Contextual Information: e.g. MD08
Wednesday, 27 October 2010
Contextual Information: e.g. MD08
Wednesday, 27 October 2010
PS1 Catalogue Ingester - Algorithm Implementation
•
•
Search field algorithms descended from single abstract class
“Factory” class determines which algorithm to use on PS1
object catalogue ingest as they are processed
Wednesday, 27 October 2010
Wednesday, 27 October 2010
Pre-Ingest Cuts - “Bad” IPP Detection Bitmask
SOURCE_BAD_MASK = 1111 0000 0000 0111 1111 1111 1001 1000
PM_SOURCE_MODE_...
MSB
OFF_CHIP
ON_GHOST
ON_SPIKE
SIZE_SKIPPED
X
X
X
X
X
X
X
X
X
BELOW_MOMENTS_SN
SKYVAR_FAILURE
SKY_FAILURE
MOMENTS_FAILURE
EXT_LIMIT
CR_LIMIT
SATURATED
DEFECT
BADPSF
EXTERNAL
BLEND
SATSTAR
X
X
POOR
FAIL
X
X
X
LSB
Wednesday, 27 October 2010
Post-Ingest Automated Cuts
Wednesday, 27 October 2010
Post-Ingest Automated Cuts
1. Set Quality Flag for each object where:
XY moments of objects < 1.2
PSF instrumental magnitude error < 0.22
PSF instrumental magnitude error > 0.0
Wednesday, 27 October 2010
Post-Ingest Automated Cuts
1. Set Quality Flag for each object where:
XY moments of objects < 1.2
PSF instrumental magnitude error < 0.22
PSF instrumental magnitude error > 0.0
2. Promote objects to Eyeball List where:
RMS position scatter < 0.5 arcsec
Quality detections >= 3 within 7 observation window
Quality detection filters >= 2
Wednesday, 27 October 2010
Post-Ingest Automated Cuts
1. Set Quality Flag for each object where:
XY moments of objects < 1.2
PSF instrumental magnitude error < 0.22
PSF instrumental magnitude error > 0.0
2. Promote objects to Eyeball List where:
RMS position scatter < 0.5 arcsec
Quality detections >= 3 within 7 observation window
Quality detection filters >= 2
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Wednesday, 27 October 2010
Post-Ingest Automated Cuts
1. Set Quality Flag for each object where:
XY moments of objects < 1.2
PSF instrumental magnitude error < 0.22
PSF instrumental magnitude error > 0.0
2. Promote objects to Eyeball List where:
RMS position scatter < 0.5 arcsec
Quality detections >= 3 within 7 observation window
Quality detection filters >= 2
observation
window = 7
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
Wednesday, 27 October 2010
Post-Ingest Automated Cuts
1. Set Quality Flag for each object where:
XY moments of objects < 1.2
PSF instrumental magnitude error < 0.22
PSF instrumental magnitude error > 0.0
2. Promote objects to Eyeball List where:
RMS position scatter < 0.5 arcsec
Quality detections >= 3 within 7 observation window
Quality detection filters >= 2
observation
window = 7
quality detections = 00000010010110
observation filters = yzzgrirgiizgry
quality detections in 3 filters
Wednesday, 27 October 2010
Python Web Interface
Template
candidate.html
<tr>
{% for column in table.columns %}
<th>
{% if column.sortable %}
<a href="?sort={{ column.name_toggled }}">
{{ column }}
</a>
{% if column.is_ordered_reverse %}
<img src="/ps1/site_media/images/up.jpg" />
{% else %}
<img src="/ps1/site_media/images/down.jpg" />
{% endif %}
{% else %}
{{ column }}
{% endif %}
</th>
{% endfor %}
</tr>
View
views.py
def candidate(request, tcs_transient_objects_id):
transient = get_object_or_404(TcsTransientObjects, pk=tcs_transient_objects_id)
initial_queryset = WebViewRecurrentObjectsPresentation.objects.filter(transient_object_id = transient.id)
table = WebViewRecurrentObjectsPresentationTable(initial_queryset, order_by=request.GET.get('sort', 'mjd_obs'))
return render_to_response('psdb/candidate.html',{'transient' : transient, 'table' : table})
Model
models.py
class WebViewRecurrentObjectsPresentation(models.Model):
id = models.IntegerField(primary_key=True, db_column='id')
transient_object_id = models.IntegerField(db_column='transient_object_id')
mjd_obs = models.FloatField(db_column='mjd_obs')
RA = models.FloatField(db_column='ra_psf')
DEC = models.FloatField(db_column='dec_psf')
mag = models.FloatField(db_column='mag')
filter = models.CharField(max_length=90, db_column='filter')
flags = models.CharField(max_length=90, db_column='flags')
cmf_file = models.CharField(max_length=90, db_column='cmf_file')
image = models.CharField(max_length=90, db_column='filename')
class Meta:
db_table = u'psdb_web_v_recurrent_objects_presentation'
@property
def flags_bin(self):
a = bin(int(str(self.flags)),32)
return a
Wednesday, 27 October 2010
Web Interface - Human Cuts
Wednesday, 27 October 2010
Wednesday, 27 October 2010
Wednesday, 27 October 2010
Wednesday, 27 October 2010
Wednesday, 27 October 2010
Horror Show
Bad Subtraction Kernel
Parameters
Bad Template
Bad Registration - Dipoles
Bright Star artifacts
Wednesday, 27 October 2010
0D07aak
SN Ic, z=0.116
Target
Wednesday, 27 October 2010
Reference
Difference
Crossmatching with CfA Objects
Wednesday, 27 October 2010
~ 100 Spectroscopically Confirmed SNe
Wednesday, 27 October 2010
PS1 MD Data Processing - 25/10/2010
PS1 Catalogue Files Processed (3.1GB)
Total Objects Processed
Rejected Objects (bad flags)
Unique Objects ingested
Recurrent Objects ingested
Total Thread Time (seconds)
39,714
13,299,196
4,020,614
3,941,472
5,337,110
1,398,243
Daily Object Processing Capacity (8 Threads) ~ 6,500,000
Wednesday, 27 October 2010
PS1 MD Data Processing 2010
Unique Objects
3,941,472
Object Type
Orphan
Variable Star
Nuclear Transient
AGN
Supernova
Wednesday, 27 October 2010
Number
2,483,784
1,058,433
195
1,609
397,451
PS1 MD Data Processing 2010
Objects passing automated quality thresholds
Unique Objects
26,314
Object Type
Orphan
Variable Star
Nuclear Transient
AGN
Supernova
Wednesday, 27 October 2010
Number
1,366
20,441
6
176
4,218
PS1 MD Data Processing 2010
Objects further refined by human examination
(Confirmed/Good/Possible Objects)
Unique Objects
1,359
Object Type
Orphan
Variable Star
Nuclear Transient
AGN
Supernova
Wednesday, 27 October 2010
Number
298
255
0
112
694
MD Transient Server Run Time
•
•
•
•
•
•
•
Download Catalogues (10 minutes)
Download GPC1 Database (10 minutes to 3 hours)
Backup Ingest DB (45 minutes)
Organise files for ingest & Run Classifier (3.5 hours)
Run Post Ingest cuts (40 minutes)
Request Postage stamps (5 minutes)
Download Postage stamps (3-10 hours)
Wednesday, 27 October 2010
Outlook for MD Survey
• Implement Dipole Stats and other new columns in pre
and post-Post Ingest Cuts
• Re-examine pre-ingest flag cuts & examine new flags in
latest IPP release
• Include more catalogues for crossmatching
• Convergence with CfA Transient Server
Wednesday, 27 October 2010
Survey Modes : 3π
Wednesday, 27 October 2010
Survey Modes : 3π
+90o
• Survey Area = 30,000
sqr. deg.
•
•
•
•
•
avg 1,600 sqr. deg. / night
30 sec exposures
~220 exposures/night
1 - 2 Tb/night
o
-30
Whole sky 4 times a
year in five filters g,r,i,z,Y
Wednesday, 27 October 2010
Survey Modes : 3π
Wednesday, 27 October 2010
Survey Modes : 3π
Difference Catalogue forecast
Based on Medium Deep stats so far:
• ~ 220 Object Catalogues produced per night in grizY (1.5 GB/day)
• ~ 16,000 objects / exposure
• ~ 3,500,000 rows per day to process
• ~ 9GB of postage stamp image data / day
Wednesday, 27 October 2010
Survey Modes : 3π
3π Transient Searching
• Difference imaging with static sky – start end
of 2010 start of 2011 ?
• What can we do before then ?
• 3 Faint Galaxy SN survey :
• Started in May 2010:
• 7 SNe and 4 AGNs
Wednesday, 27 October 2010
15 good candidates (spectroscopic follow-up):
Survey Modes : 3π
3π Faint Galaxy Supernova Survey
• ~ 220 Object Catalogues produced per night in griz
•97% “magicked”
•35-40% in Sloan areas
•200,000 objects per catalogue file (i.e. per exposure)
•50,000 after pre-ingest optimisation
• No differencing done
• Download & crossmatch 6GB/day object catalogues
•(33, 40, now 55 columns of info per FITS table x 60 + header)
• FGSS database still being developed - no growth stats yet...
• Download & store 20MB/day “postage stamp” images
Wednesday, 27 October 2010
3π Faint Galaxy Supernova Survey
- Download all the 3π-catalogs produced by IPP (smf files)
- Select Sloan fields and calibrate the PS1 magnitudes to the Sloan magnitude.
- Cross-correlate all the good* sources with the SDSS catalogs and select all the sources
for which the closer SDSS object is classified as a galaxy and falls within 3 arcsec of
the PS1-3π detection.
- Extract all PS1 detections that meet the following criteria:
15 < PS1 mag (observed filter) < 20
19 < sloan mag (r) < 23
distance (sloan galaxy - PS1 detection) < 3 arcsec
PS1 mag 2 mag brighter than sloan mag (in the obs. filter)
occurrences >= 3 and 2 arcsec grouping of detections
Wednesday, 27 October 2010
FGSS
SNe
4 Termonuclear SNe
3 Core Collapse (IIL,IIP,II)
AGN
Wednesday, 27 October 2010
FGSS crossmatching
• Cross-match computation done in memory and achieved by dividing the candidates
into subsets
• (chip by chip) (optimise the cross-match) (140,000 Sloan galaxies per catalogue file)
• each spatial cross-match among one catalogue file and the sloan galaxies in the same
area selects:
• r ~ 524 crossmatches i ~ 870 crossmatches
• g ~ 345 crossmatches
z ~ 348 crossmatches
• ~ 0.7 good detection / each catalogue file
(Magnitude and distance cut)
• 171 candidates (recurrence cut and grouping of detections)
• ~ 10% of 171 selected for observations:
– ~ 5% spectroscopically classified as SNe
– ~ 2.5 % spectroscopically classified as AGN
Wednesday, 27 October 2010
PS1 detections - Sloan catalog
•
Wednesday, 27 October 2010
Query Sloan chip by chip
Running Time
•
Download files: 9h / day ?
~215 smf files per night
~208 smf files magiked
208 x (140s-180s) / smf file (30.5 M) ~ 8h-10h
•
Cross-match: 10h / day ?
•
Zero point - sloan area check
~ 208 x time sloan field check = 30m - 1h30m
•
(38%) in the sloan area
~ 80 x time cross-match =
•
<1h>
Other steps:
(4h - 17h)
< 9h >
3h/month ?
•
Selection cut + web-page (without stamps) 30m
•
Stamp servers request-download (~ 200 stamps per months) ???
•
Eyeballing (50 candidates per months)
•
Download extra stamps (5-10 candidates per month ~150 stamps) ?
Wednesday, 27 October 2010
1h
2-5 good candidates per month
Outlook for 3π Survey
•
•
•
•
•
•
Fully implement FGSS database
• Crossmatching efficiency may need to improve
3π diffs should start to flow in December/January
• This may replace the FGSS, or the two may run in parallel
for a time
Improve pre-ingest cuts - this reduces CPU time in
crossmatching algorithms (but at expense of not ingesting the
data row)
Tune the code to minimise unnecessary latency (e.g.
instantiation of unnecessary classes)
Liaise with Networks staff about improving data links to server
Throw more processors at the problem
• Most latency so far is CPU bound
Wednesday, 27 October 2010
Wednesday, 27 October 2010