PanSTARRS1 :
Transient source identifications and database
management
D. Young, K. Smith, S.J. Smartt and QUB SN group
Michael Wood-Vasey (Pittsburgh), CfA Harvard
Wednesday 20 January 2010
PanSTARRS1 Overview
• 1.8m telescope
• 1.4 Gigapixel Camera
• 7 square deg FOV
• Image Processing Pipeline (IPP)
• Few Tb of data every night
Wednesday 20 January 2010
PanSTARRS1 Overview
PanSTARRS UK:
• Durham University
• University of
Edinburgh
• Queen’s University
Belfast
Wednesday 20 January 2010
Survey Modes : 3π
Wednesday 20 January 2010
Survey Modes : 3π
+90o
• Survey Area = 30,000
sqr. deg.
• 6,000 sqr. deg. / night
• Whole sky 4 times a
year in five filters g,r,i,z,Y
o
-30
Wednesday 20 January 2010
Survey Modes : Medium Deep
Wednesday 20 January 2010
Survey Modes : Medium Deep
•10 x 7 squ. deg. fields
•240 sec exposures
•Observed in g,r,i,z,Y
every 4 nights
•Well known fields
Wednesday 20 January 2010
Searching for core-collapse
supernovae with PS1
Table 7. Estimated intrinsic and detectable CCSRs from the SDSS DR5 HSFG, LSFG and Omitted Galaxies within z = 0.04 using PS1. The CCSRs for the HSFG and LSFG samples are determined
using galaxy Hα luminosities, whereas the rates for the sample of nearby luminous galaxies omitted from SDSS DR5 spectroscopic survey are determined from their U-band luminosities
HSFGs
LSFGs
Omitted Galaxies
Total Sample
Detected CCSNe yr−1
12+log(O/H)
Galaxies
CCSNe yr−1
Galaxies
CCSNe yr−1
Galaxies
CCSNe yr−1
Galaxies
CCSNe yr−1
SDSS Area
PS1 Area
No Limit
18 350
115.6
6 000
34.1
1 216
12.7
25 091
162.4
108.6
570.4
< 8.4
8 019
13.2
1 757
0.8
-
-
9 776
13.9
10.3
54.1
< 8.3
4 290
6.9
1 025
0.3
-
-
5 315
7.2
5.3
28.0
< 8.2
1 713
3.1
401
0.1
-
-
2 114
3.2
2.4
12.5
< 8.1
537
1.0
116
0.0
-
-
653
1.1
0.8
4.3
Young et al. (2008)
Wednesday 20 January 2010
Searching for core-collapse
supernovae with PS1
Table 7. Estimated intrinsic and detectable CCSRs from the SDSS DR5 HSFG, LSFG and Omitted Galaxies within z = 0.04 using PS1. The CCSRs for the HSFG and LSFG samples are determined
using galaxy Hα luminosities, whereas the rates for the sample of nearby luminous galaxies omitted from SDSS DR5 spectroscopic survey are determined from their U-band luminosities
HSFGs
LSFGs
Omitted Galaxies
Total Sample
Detected CCSNe yr−1
12+log(O/H)
Galaxies
CCSNe yr−1
Galaxies
CCSNe yr−1
Galaxies
CCSNe yr−1
Galaxies
CCSNe yr−1
SDSS Area
PS1 Area
No Limit
18 350
115.6
6 000
34.1
1 216
12.7
25 091
162.4
108.6
570.4
< 8.4
8 019
13.2
1 757
0.8
-
-
9 776
13.9
10.3
54.1
< 8.3
4 290
6.9
1 025
0.3
-
-
5 315
7.2
5.3
28.0
< 8.2
1 713
3.1
401
0.1
-
-
2 114
3.2
2.4
12.5
< 8.1
537
1.0
116
0.0
-
-
653
1.1
0.8
4.3
Young et al. (2008)
570.4
CCSNe/yr
Wednesday 20 January 2010
Searching for core-collapse
supernovae with PS1
Table 7. Estimated intrinsic and detectable CCSRs from the SDSS DR5 HSFG, LSFG and Omitted Galaxies within z = 0.04 using PS1. The CCSRs for the HSFG and LSFG samples are determined
using galaxy Hα luminosities, whereas the rates for the sample of nearby luminous galaxies omitted from SDSS DR5 spectroscopic survey are determined from their U-band luminosities
HSFGs
LSFGs
Omitted Galaxies
Total Sample
Detected CCSNe yr−1
12+log(O/H)
Galaxies
CCSNe yr−1
Galaxies
CCSNe yr−1
Galaxies
CCSNe yr−1
Galaxies
CCSNe yr−1
SDSS Area
PS1 Area
No Limit
18 350
115.6
6 000
34.1
1 216
12.7
25 091
162.4
108.6
570.4
< 8.4
8 019
13.2
1 757
0.8
-
-
9 776
13.9
10.3
54.1
< 8.3
4 290
6.9
1 025
0.3
-
-
5 315
7.2
5.3
28.0
< 8.2
1 713
3.1
401
0.1
-
-
2 114
3.2
2.4
12.5
< 8.1
537
1.0
116
0.0
-
-
653
1.1
0.8
4.3
Young et al. (2008)
570.4
~760
SNe/yr
CCSNe/yr
Wednesday 20 January 2010
PS1 Transients
Wednesday 20 January 2010
PS1 Transients
Search
Template Difference
Supernovae
Variable stars
AGN
Diffraction spikes
Dipoles
Wednesday 20 January 2010
Images from SDSS II SN Survey.
Sako et al. (2008)
Contextual Information : e.g. MD08
Wednesday 20 January 2010
Contextual Information : e.g. MD08
Wednesday 20 January 2010
Contextual Information : e.g. MD08
Wednesday 20 January 2010
Contextual Information : e.g. MD08
Wednesday 20 January 2010
Contextual Information : e.g. MD08
Wednesday 20 January 2010
Contextual Information : e.g. MD08
Wednesday 20 January 2010
PS1 Transient Classification Server
Wednesday 20 January 2010
Point Source Catalogues :
SDSS, GSC v2.3, 2MASS PSC
Variable Star
3.0 arcsec
Star cluster Westerlund 1
Wednesday 20 January 2010
Quasars and Active Galactic Nuclei :
Veron et al. (2006), Kauffmann et al. (2003), NED
AGN
2 arcsec
Core of NCG 4261
Hubble Space Telescope
Wide Field Planetary Camera
Wednesday 20 January 2010
Galaxy Catalogues
NED, SDSS galaxies, 2MASS XSC ...
15 Kpc
15 Kpc
Supernova
15 Kpc
15 Kpc
Abell galaxy cluster
Credit : NASA, ESA, Hubble Heritage Team
(STScI / AURA)
Wednesday 20 January 2010
Galaxy Catalogues
NED, SDSS galaxies, 2MASS XSC ...
Supernova
3π = 8.2 arcsec
MD = 3.4 arsec
Abell galaxy cluster
Credit : NASA, ESA, Hubble Heritage Team
(STScI / AURA)
Wednesday 20 January 2010
?
Wednesday 20 January 2010
Orphan
PanSTARRS1 :
QUB Transient Classification Server Implementation
Wednesday 20 January 2010
Predicted PS1 Catalogue Data Volume
Transient
Object
Catalogue
Predicted
Nightly
Number
of
Transients
Storage
Required
per
Transient
Units
50,000
3 KB
Nightly
Storage
Required
150 MB
Yearly
Storage
Required
55 GB
Total
Storage
over
3.5
Years
Wednesday 20 January 2010
Data
Volume
193 GB
Predicted Followup Data Volume
Instrument
WHT
Value
Units
42 nights
10 GB
per
night
Gemini
490 hours
1 GB
per
hour
Wednesday 20 January 2010
NOT
0.42 TB
LT
0.42 TB
Total
Storage
required
over
3.5
Years
1.75 TB
Predicted PS1 Image Data Volume
Transient
Object
Images
Predicted
Nightly
Number
of
Transients
Predicted
Large
FITS
images
required
3
arcmin
FITS
data
size
Units
50,000
10
4 MB
FITS
images
per
object
(pre/post/diff
x
2
x
6
epochs)
36
Predicted
Small
JPEG
images
required
400
1
arcmin
JPEG
data
size
0.1 MB
JPEG
images
per
object
(pre/post/diff
x
2)
Wednesday 20 January 2010
Data
Volume
6
Traffic
per
night
1.7 GB
Yearly
Storage
Required
(70%
nights
used)
430 GB
Total
Storage
over
3.5
Years
1.5 TB
Hardware
Transient Server & Local IPP
DELL PowerEdge 2950 III (64GB)
8 Core Xeon X5460 (3.16 GHz)
1 TB (RAID 5) Internal SAS Storage
Transient Server Database Logs
& Temporary Tables
Web & File Server
DELL PowerEdge 2950 III (2GB)
4 Core Xeon E5405 (2.0 GHz)
DELL PowerVault MD3000
5.3 TB (RAID 5) SAS Storage Array
Transient Server Database Tables
DELL PowerVault MD1000
11.2 TB (RAID 5) SATA Storage Array
Database Backup Staging
Image Stamps
Follow-up Data
Wednesday 20 January 2010
DELL PowerVault MD1000
11.2 TB (RAID 5) SATA Storage Array
Backup Mirror
IPP Data Products
External Catalogue Ingestion
8 Generic Catalogues
> 20 Medium Deep Field Catalogues
109
2x
rows
0.5 TBytes (MyISAM)
> 40 views
105
3x
rows
0.1 TBytes (MyISAM)
MD01 NED
Veron 12
MD05 NED
MD01 Chiappetti 2005
2MASS XSC
MD05 Brunner 2008
MD06 NED
MD01 Pierre 2007
SDSS Stars
MD02 NED
2MASS PSC
MD07 NED
MD02 Giacconi 2002
Guide Star 2.3
MD07 Laird 2009
MD02 LeFevre 2004
NED
SDSS Photo Galaxies
MD07 Nandra 2005
MD02 Leher 2005
MD08 NED
MD02 Virani 2006
SDSS Galaxies
MD08 Manners 2003
MD03 NED
MD09 NED
MD04 NED
MD10 NED
MD04 Hasinger 2007
MD04 Trump 2007
• Source data in various formats (CSV, FITS,VO Tables)
• Custom ingest scripts written (PERL, C++, Java)
• Policy adopted of NaN, ±∞, null values = NULL
• Unit sphere cartesian coords (cx, cy, cz) calculated on ingest for cone searches
• Indexing based on JHU HTM
Wednesday 20 January 2010
Crossmatching: JHU Hierarchical Triangular Mesh
e.g. Level 20 HTM id for RA, DEC = 243.334412243, 53.9783926374
N112212112030032100002
(1474714251366610)
Wednesday 20 January 2010
HTM vs Cartesian Cone Search
17
HTM
Cartesian
16.5
16
15.5
15
14.5
14
8
Wednesday 20 January 2010
8.5
9
9.5
10
10.5
11
11.5
12
HTM “circleRegion” + Cone Search
SELECT
*
FROM
catalogue
WHERE
(
htm_id BETWEEN id(0) AND id(1)
OR
htm_id BETWEEN id(2) AND id(3)
OR
.
.
OR
htm_id BETWEEN id(n-1) AND id(n)
)
AND (
cx * object x coordinate +
cy * object y coordinate +
cz * object z coordinate >= cos(radius in radians)
);
Wednesday 20 January 2010
Software Components
PS1 Image Processing
Pipeline
HTTP
PS1 Postage Stamp
Server
Command
Line/CRON
Web
Interface
PERL
Wrapper
Django
Framework
C++
Ingester
/
Classifier
Apache
Python
PS1
Image
mod_python Request/Fetch
CCFits
HTM
MySQL++
MySQLDB
Python
API
MySQL
API
MySQL
Database
CFITSIO
PS1
Difference
Catalogues
(FITS)
CentOS
5
(RHEL
5)
Wednesday 20 January 2010
HTTP
PS1
Image
Stamps
'(
))
%&
%&
'*%+,
'*%+,
2
!%&
3(-
'%&
-
'(
))
#
! *%+,
((
))
(#- (./++
##-
%&
Transient
Server
Database
Schema
'*%+,
'*%+,
#2
)
!('
#'(
)
'%&
'(
)
#
! *%+,
'(#
0('
'(#
)
'((1-2
##-
((
))
##-
%&
(',/,
!('./++
)
!%&
./++
(
"
(#- !
"
./++
(#- (./++
!%&
#
4
#
$
%&
!%&
!%&
!
"
Wednesday 20 January 2010
!%&
$%
&*$+,
&*$+,
$%
&$%
"
''
())
""-
Transient
Server
Database
Schema
"2
)
"&'
)
&'
)
*$+,
""-
$%
&'
())
*$+,
&'
())
$%
&*$+,
&*$+,
'&
&$%
"
0'&
&''1-2
)
''
())
""-
&'"
&'"
)
'"- !
./++
'"-'./++
$%
$%
'&,/,
'&./++
$%
"
3
"
#
Wednesday 20 January 2010
Implemented
Classification
Algorithm
Process Transients
Incoming
Transient
General
Processing
Extract Transient Info
Existing Catalogue
Search
Store Transient Metadata
Search Existing Transients - 1.0"
Match
Store Transient Recurrent Observations
Cross-Match
recording
No Match
Search Veron - 2"
Classification Flags
Match
Record Veron Cross-Matches
Match
Record 2MASS XSC Cross-Matches
Match
Record SDSS Stars Cross-Matches
No Match
Search 2MASS XSC - 2"
Flag as AGN
No Match
Search SDSS stars - 3"
Some of the objecs in 2MASS PSC and Guide
Star are actually galaxies. Double-check
No Match
Search 2MASS PSC - 3"
Match
PSC SubSearch SDSS Galaxies - 3"
Record 2MASS PSC Cross-Matches
Flag as Variable Star
No Match
Match
No Match
Set VariableStar Mismatch Flag
Not Set
Test VariableStar Mismatch Flag
Set
Search GuideStar - 3"
Match
GS SubSearch SDSS Galaxies - 3"
No Match
Search NED - 50'
Record GuideStar Cross-Matches
No Match
Match
Match
Iterate through all NED matches
Calculate Separations from NED sources
Extract NED redshift info
Does not exist
Exists
Check Within 15kpc of NED Source
No Match
No Match
Match
Check within 2" of NED Source
No Match
Check within 1' of NED source
No Match
Search SDSS galaxies - 5"
Match
Match
Match
Record NED 35kpc Cross-Matches
Record NED 2" Cross-Matches
Record NED 1' Cross-Matches
Flag as SN
Record SDSS Galaxies Cross-Matches
No Match
Flag as Orphan
Ingested
Catalogues
Store Transient Object
Newly Created
Transient Catalogue
Wednesday 20 January 2010
Store Cross-Matches
PS1 Catalogue Ingester - Algorithm Implementation
•
•
Search field algorithms descended from single abstract class
“Factory” class determines which algorithm to use on PS1
object catalogue ingest as they are processed
Wednesday 20 January 2010
Python Web Framework
Template
candidate.html
<tr>
{% for column in table.columns %}
<th>
{% if column.sortable %}
<a href="?sort={{ column.name_toggled }}">
{{ column }}
</a>
{% if column.is_ordered_reverse %}
<img src="/ps1/site_media/images/up.jpg" />
{% else %}
<img src="/ps1/site_media/images/down.jpg" />
{% endif %}
{% else %}
{{ column }}
{% endif %}
</th>
{% endfor %}
</tr>
View
views.py
def candidate(request, tcs_transient_objects_id):
transient = get_object_or_404(TcsTransientObjects, pk=tcs_transient_objects_id)
initial_queryset = WebViewRecurrentObjectsPresentation.objects.filter(transient_object_id = transient.id)
table = WebViewRecurrentObjectsPresentationTable(initial_queryset, order_by=request.GET.get('sort', 'mjd_obs'))
return render_to_response('psdb/candidate.html',{'transient' : transient, 'table' : table})
Model
models.py
class WebViewRecurrentObjectsPresentation(models.Model):
id = models.IntegerField(primary_key=True, db_column='id')
transient_object_id = models.IntegerField(db_column='transient_object_id')
mjd_obs = models.FloatField(db_column='mjd_obs')
RA = models.FloatField(db_column='ra_psf')
DEC = models.FloatField(db_column='dec_psf')
mag = models.FloatField(db_column='mag')
filter = models.CharField(max_length=90, db_column='filter')
flags = models.CharField(max_length=90, db_column='flags')
cmf_file = models.CharField(max_length=90, db_column='cmf_file')
image = models.CharField(max_length=90, db_column='filename')
class Meta:
db_table = u'psdb_web_v_recurrent_objects_presentation'
@property
def flags_bin(self):
a = bin(int(str(self.flags)),32)
return a
Wednesday 20 January 2010
Web Interface
Wednesday 20 January 2010
Web Interface
Wednesday 20 January 2010
June PS1 MD 06/07/08 Data Processing
PS1
Catalogue
Files
Processed
Total
Objects
Processed
Rejected
Objects
Unique
Objects
ingested
Recurrent
Objects
ingested
Processing
Time
(8
threads)
Daily
Object
Processing
Capacity
Wednesday 20 January 2010
3487
1,027,097
541,282
396,383
89,432
3
hours
~
8,000,000
June 2009 Medium Deep 07/08/09:
“Bad” Flag Filtered
Unique
Objects
396,383
Object
Type
Orphan
Variable
Star
AGN
SN
Wednesday 20 January 2010
Number
170,825
83,085
1,557
140,916
June 2009 Medium Deep 07/08/09:
“Bad” Flag Filtered
Occurrences >= 4
Unique
Objects
9,837
Object
Type
Orphan
Variable
Star
AGN
SN
Wednesday 20 January 2010
Number
121
7,989
173
1,554
CBET 1988
SN 2009kb
SN 2009kd
SN 2009ke SN 2009kf SN 2009kg
SN 2009kh SN 2009ki
SN 2009kj
Wednesday 20 January 2010
To Do:
• Algorithm enhancement (reduce false positives)
• New catalogue integration, parameter tuning
• Convergence with other Transient Servers (e.g. CfA)
• Automated catalogue file downloads
• Event triggers / alerts
• Web interface updates (user voting, comments, etc)
• Supply candidate image triplets to SN Zoo
Wednesday 20 January 2010
Wednesday 20 January 2010
Choosing the Cone Search Radius
Wednesday 20 January 2010
Wednesday 20 January 2010
PS1
Supernova
Survey
Web
Interface
Wednesday 20 January 2010
PS1 IPP Object Catalogue Bitmasks
Flag Name
Binary representation
Hex
Decimal
Definition
------------------------------------------------------------------------------------------------------------------------PSFMODEL
0000 0000 0000 0000 0000 0000 0000 0001
00 00 00 01
1
Fit with PSF model
EXTMODEL
0000 0000 0000 0000 0000 0000 0000 0010
00 00 00 02
2
Fit with extended model
FITTED
0000 0000 0000 0000 0000 0000 0000 0100
00 00 00 04
4
Fit with non-linear model
FAIL
0000 0000 0000 0000 0000 0000 0000 1000
00 00 00 08
8
Non-linear fit failed
POOR
0000 0000 0000 0000 0000 0000 0001 0000
00 00 00 10
16
Non-linear fit poor
PAIR
0000 0000 0000 0000 0000 0000 0010 0000
00 00 00 20
32
Fit with double PSF
PSFSTAR
0000 0000 0000 0000 0000 0000 0100 0000
00 00 00 40
64
Used to define PSF model
SATSTAR
0000 0000 0000 0000 0000 0000 1000 0000
00 00 00 80
128
Model peak is above saturation
BLEND
0000 0000 0000 0000 0000 0001 0000 0000
00 00 01 00
256
Blended with other sources
EXTERNAL
0000 0000 0000 0000 0000 0010 0000 0000
00 00 02 00
512
Based on supplied input position
BADPSF
0000 0000 0000 0000 0000 0100 0000 0000
00 00 04 00
1024
Unable to estimate object PSF
DEFECT
0000 0000 0000 0000 0000 1000 0000 0000
00 00 08 00
2048
Suspected defect
SATURATED
0000 0000 0000 0000 0001 0000 0000 0000
00 00 10 00
4096
Suspected saturated (bleed trail)
CR_LIMIT
0000 0000 0000 0000 0010 0000 0000 0000
00 00 20 00
8192
Suspected cosmic ray
EXT_LIMIT
0000 0000 0000 0000 0100 0000 0000 0000
00 00 40 00
16384
Suspected extended
MOMENTS_FAILURE
0000 0000 0000 0000 1000 0000 0000 0000
00 00 80 00
32768
Failed to measure moments
SKY_FAILURE
0000 0000 0000 0001 0000 0000 0000 0000
00 01 00 00
65536
Failed to measure local sky
SKYVAR_FAILURE
0000 0000 0000 0010 0000 0000 0000 0000
00 02 00 00
131072
Failed to measure sky variance
BELOW_MOMENTS_SN 0000 0000 0000 0100 0000 0000 0000 0000
00 04 00 00
262144
Moments not measured due to low
BIG_RADIUS
0000 0000 0001 0000 0000 0000 0000 0000
00 10 00 00
1048576
Small radius has poor moments
AP_MAGS
0000 0000 0010 0000 0000 0000 0000 0000
00 20 00 00
2097152
Measured aperture magnitude
BLEND_FIT
0000 0000 0100 0000 0000 0000 0000 0000
00 40 00 00
4194304
Fit as a blend
EXTENDED_FIT
0000 0000 1000 0000 0000 0000 0000 0000
00 80 00 00
8388608
Fit with full extended fit
EXTENDED_STATS
0000 0001 0000 0000 0000 0000 0000 0000
01 00 00 00
16777216
Calculated extended aperture st
LINEAR_FIT
0000 0010 0000 0000 0000 0000 0000 0000
02 00 00 00
33554432
Fit with linear fit
NONLINEAR_FIT
0000 0100 0000 0000 0000 0000 0000 0000
04 00 00 00
67108864
Fit with non-linear fit
RADIAL_FLUX
0000 1000 0000 0000 0000 0000 0000 0000
08 00 00 00
134217728
Calculated radial flux measuremen
SIZE_SKIPPED
0001 0000 0000 0000 0000 0000 0000 0000
10 00 00 00
268435456
Could not be determine size
ON_SPIKE
0010 0000 0000 0000 0000 0000 0000 0000
20 00 00 00
536870912
Peak lands on diffraction spike
ON_GHOST
0100 0000 0000 0000 0000 0000 0000 0000
40 00 00 00
1073741824
Peak lands on ghost or glint
OFF_CHIP
1000 0000 0000 0000 0000 0000 0000 0000
80 00 00 00
2147483648
Peak lands off edge of chip
Flag Name
Binary representation
Hex
Decimal
Definition
------------------------------------------------------------------------------------------------------------------------BAD_FLAGS
1111 0000 0000 0111 1111 1111 1001 1000
F0 07 FF 98
4027056024
See above
GOOD_FLAGS
0000 0000 0000 0000 0000 0000 0000 0101
00 00 00 05
5
See above
Wednesday 20 January 2010
HTM + Cone Search C++ Implementation
vector<htmRange> htmList=htm.circleRegion(ra, dec, radius/60.0);
query << setprecision(18) << "select " << columnList << " from " << table_name
<< " where (htm" << level << "ID between ";
for(size_t idx=0; idx<htmList.size(); idx++)
{
if (idx != 0) query << "or htm" << level << "ID between ";
query << htmList[idx].lo << " and " << htmList[idx].hi << endl;
}
double cosDec =
query << ") and
query << ”
cx
query << " + cy
query << " + cz
query << " >= "
query << ")";
query << ";";
Wednesday 20 January 2010
cos(dec*RADIANS);
(";
* " << cos(ra*RADIANS) * cosDec;
* " << sin(ra*RADIANS) * cosDec;
* " << sin(dec*RADIANS);
<< cos((radius/3600)*RADIANS);