[HSC-1259] Add variance to HSC output catalog Created: 15/May/15
Updated: 19/Oct/15 Resolved:
19/Oct/15
Status:
Project:
Component/s:
Affects
Version/s:
Fix Version/s:
Done
HSC Data Management
hscPipe
3.x
Type:
Reporter:
Resolution:
Labels:
Remaining
Estimate:
Time Spent:
Original
Estimate:
Improvement
Alexie
Done
None
Not Specified
Attachments:
check_var_ellipse.py
var_cat_src_100x100.png
var_cat_src_20x20.png
var_cat_src_50x50.png
var_cat_src_ellipse_with_scaleVariance.png
var_cat_src_price.png
Hisanori Furusawa
Winter 2016 development 1
Reviewers:
Sprint:
None
Priority:
Assignee:
Votes:
Major
Paul Price
0
Not Specified
Not Specified
Description
Can we please record the variance for each object and add that to the output? We'd like to use
this to generate random catalogs as discussed at the weak lensing HSC meeting.
Comments
Comment by Paul Price [ 15/May/15 ]
What do you mean by "the variance for an object"? Do you mean the single pixel variance at the
position of the center? Or do you mean the sum of the variance over some aperture? Or
something else?
Comment by Alexie [ 15/May/15 ]
The central pixel value I think would be fine. We'll also record a pixel value when drawing
random points. Robert suggested the variance didn't vary much over the size of an object
anyway. So taking a central pixel value would not be so different than taking a mean over an
aperture?
Comment by Jim Bosch [ 15/May/15 ]
I'd recommend averaging over the Footprint. That won't usually be very different from the value
at the peak, but it's probably a bit better and just as easy to implement.
Comment by Paul Price [ 15/May/15 ]
It seems to me that the central pixel values are recorded in the image (in the variance plane),
while averages can probably be derived from the aperture flux error and the footprint size. So
I'm not sure we need to provide anything new in the catalogs.
Comment by Alexie [ 15/May/15 ]
For the randoms, we would be recording a pixel value. So to be consistent, a pixel value might
be more appropriate. Or if we average over an aperture, the perhaps a fixed aperture so that we
can do the same thing in the randoms? Or maybe it won't make a difference? You would know
better than I do ...
Paul: we'd like to add this to the output catalogs because we'd like to have this added to the
object database.
Comment by Alexie [ 02/Jun/15 ]
Actually, Jean pointed out to me that the variance plane also includes the poisson variance due
to the object itself which we do not want to include. So I think what we need to do is to define
some radius R and compute the mean variance within R after excluding all pixels that belong to
any footprint. We'd like R to be larger than the sizes of bright galaxies but smaller than the
typical size of a CCD.
Comment by Paul Price [ 03/Jun/15 ]
We could make a "background model" of the variance plane that would filter out the sources,
and use that to get the variance at the position of each object, but this doesn't fit into our usual
measurement stage.
Are you sure you can't derive the mean variance within the aperture from the reported aperture
flux error?
Comment by Alexie [ 04/Jun/15 ]
Making a background model that excludes sources would work for sure.
I'm not sure what you mean by using the reported aperture flux error - doesn't that include the
Poisson term from objects - which is exactly what we'd like to exclude?
Comment by Paul Price [ 02/Jul/15 ]
Furusawa-san, are you able to review this, please? (If not, please let me know and I will ask
someone else.)
There is one commit in each of meas_algorithms and obs_subaru.
price@price-laptop:~/hsc/meas_algorithms (tickets/HSC-1259=) $ git --nopager log --stat --reverse origin/master..
commit 161be63c89b05826aacf881f4b01d4e7024a8236
Author: Paul Price <price@astro.princeton.edu>
Date:
Thu Jul 2 19:26:15 2015 -0400
add measurement algorithm to report background variance
Knowing the variance will help with the fake object pipeline.
We measure the median variance in a Footprint scaled from the
object shape (default scale is 5) so the value is not terribly
influenced by the object and reflects instead the variance in
the background near the object.
include/lsst/meas/algorithms.h
include/lsst/meas/algorithms/Variance.h
python/lsst/meas/algorithms/algorithmRegistry.py
python/lsst/meas/algorithms/algorithmsLib.i
src/Variance.cc
++++++++++++++++++++++
tests/variance.py
+++++++++++++++++++++++
6 files changed, 316 insertions(+)
|
1
| 52
|
1
|
2
| 129
+
+++++++++
+
+
| 131
price@price-laptop:~/hsc/obs_subaru (tickets/HSC-1259=) $ git --no-pager
log --stat --reverse origin/master..
commit 91d2bbabee321eef5359d8ae8e40ba0c69e55dbd
Author: Paul Price <price@astro.princeton.edu>
Date:
Thu Jul 2 19:33:16 2015 -0400
config: activate measurement of variance for coadds
We want to measure the sky noise for each object, using the new
"variance" measurement algorithm.
config/forcedPhotCoadd.py | 2 +config/processCoadd.py
| 2 +2 files changed, 2 insertions(+), 2 deletions(-)
Comment by Paul Price [ 09/Sep/15 ]
The release is now out, so it won't hurt to add this now. Hisanori Furusawa, could you please
review it?
Comment by Hisanori Furusawa [ 09/Oct/15 ]
Sorry for taking long. I examined the cataloged variance by processing the
UD-COSMOS HSC-I patch "5,5" with the patched stack.py, and by comparing
the cataloged variance of each source with my external double-checking calculation
for each source. The comparison is done in two ways:
1) compare 'cataloged variance' with 'variance by my calculation within 5 x ellipse around the
sources on image just in the same way as in Variance.cc'
2) compare 'cataloged variance' with 'variance by my calculation within a simple box around the
sources on image' for reference
1) comparison with '5 x ellipse'
2-1) comparison with '100x100px (16.8") box'
2-2) comparison with '50x50px (8.4") box'
2-3) comparison with '20x20px (3.4") box'
Basically, we see that difference between the cataloged and double-check in
5 x ellipse is within 5% for most sources, which is harmless.
The appearance of the result with the box 20x20px is close to that of the ellipse.
I think the proposed code is acceptable for release,
though I have no idea of the cause of the 2.5% offset right away.


The scaleVariance function did not change the result.
There may be inconsistency due to sky-bkg subtraction in stack.py,
as my calculation does not change the sky bkg.
Do we have to add mask for variance calculation = DETECTED,
as the users seem to get the variance without contribution by
photon noise of sources?
The present code seems to set the default mask for stats of 'BAD' and 'SAT'.
Comment by Paul Price [ 19/Oct/15 ]
I've added DETECTED and DETECTED_NEGATIVE as default mask planes to ignore.
I tried your code after running processCoadd.py on a recent COSMOS coadd, and there is no
offset. Note that we don't expect an exact comparison, because your comparison code doesn't do
the calculation within an ellipse but within the bounding box of the ellipse.
Comment by Paul Price [ 19/Oct/15 ]
Rebased both meas_algorithms and obs_subaru against master, and merged.
Generated at Tue Feb 09 08:52:00 EST 2016 using JIRA 6.2.2#6258sha1:65ffb4362589622c100f6488635539584b0f7b98.