HSC Data Challenge1 - Software Test Items
Listed based on the DC1 target description on the wiki hsc-gw1:8000/wiki/HscDataChallenge1
Input Data CCD data
DataManager Data location
Remove cosmic ray
Mask & variance
Output Products MosaicedImages
Sharing One Code Set Codes
v. 1.0 - 8.10.2010
v. 2.0 - 9.07.2010
t description on the wiki hsc-gw1:8000/wiki/HscDataChallenge1
Requirement Assigned Member
104 CCDs arranged in the HSC FOV format Furusawa
Includes i<26 galaxies?
Includes USNO-B1.0 or SDSS stars?
Observing conditions are correct? (seeing=0.7", expt=300sec, readout=10,
gain=2.5-3.5, PA=0, EL=0)
Disortion and airmass are correctly added?
Flat pattern + bad pixels included?
Data format the same as Scam
sequential DET-ID is assigned 0 to 103
Initial WCS (10 arcsec offset from the real position)
No cosmic rays, no difference CCD size, focus, PSF
HSC simulated data are well structured in a defined directory trees, which
are accessed by dataManager
Can read policy files and derive a file path, camera/ccd/amp information
Loomis, Bick, Furusawa
necessary for analysis
Can collect frames to stack into a flat frame. Human hands in DC1? Yasuda, Fursawa, Bick
Can create flat frames by combining os-subtracted images by taking
Can put the resultant flat into the defined directory
Can collect frames to reducebased on the progId/frameID. Human hands
overscan counts can be subtracted line by line, and os regions and blanks
Can identify distinct bad pixels and mask them?
Can derive objects for PSF measurement Lupton
Can determine crude PSF across the CCD area, in a way the PSF can be
used in the next object detection and centroiding for astrometry stage
minimal implementation in DC1 Lupton
Mask and variance images are calculated in "debias and flatfielding (also
skysubtraction?)" and properly attached to exposure object
Does not affect photometry of bright objects Lupton
Good enough centroiding accuracy; much better than positional accuracy
of reference catalog sources.
Good enough accuacy in flux counting; much better than reference
Enough number of objects in each CCD (> 50 reasonable reference
Can use crude PSF information for detection and measurements at the
Can provide point sources for precise PSF determination
Can derive good PSF information across the CCD area Lupton
Resultant PSF can be obtained by specifying (x, y) positions
Can provide good source lists for the astrometry stage with good
centroiding by using thbe MeasurePSF results
The number of detections for typical dataset, Position and flux accuracy to
be tested and reported
Can provide matching Lists of matched pairs with reference sources
Loomis, Bick, Furusawa
containing (x,y, RA,Dec, SDSS mags, flags)
Well determined setting available for matching to succeed (at least >50
matched stars output per CCD) for every simulated data in DC1 (5shot x 5
The number of matched objects and robustness/quality of matching to be
tested and reported for typical simulated data and also for Suprime-Cam
Can fit a single TAN-SIP at a time for 104 CCDs assuming the CCD
geometrical arrangement is known, obtain WCS of each CCD
Fitting error of WCS solve is tested and reported
Can derive photometric zeropoint of each CCD by assuming mag of
reference sources are in HSC band system
Prepared for determination of zeropoint with color term by using SDSS
Fitting error in zeropoint and how magnitudes of objects can be recovered
to be tested and reported
Can collect FITS frames (+WCS) to be mosaiced by specifying
progId/filter/time range etc. Human hands also acceptable for DC1
Can collect necessary matchlist and source lists of each CCD Mineo?
sourceSet of objects in each CCDs can be available Lupton
(x,y,flux) with good enough accuracy of centroiding and flux measurement
to do mosaic
matched lists with reference sources available? Yasuda, Okura
Good WCS for each CCD provided by AstrometryWcsSolve available? Yasuda
Can do matching of sources across CCDs from different shots by using
WCS from astrometry stage?
Can determine the best geometrical alignment of each CCD by
determining TAN-SIP WCS keywords of each CCD
Fitting error and rms in geometrical alignment of CCDs to be tested and
reported for typical simulated dataset.
Processing speed is also to be investigated.
zeropoints from fluxCalib stage are to be used in DC1. Yasuda
Is WCS and zeropoint can be refined in Mosaic stage? If so, those results
can be re-distributed to each CCD?
Does not affect profiles and fluxes of bright sources (S/N>10) Lupton
Can subtract typical smooth sky bkg of simulated data and the resultant
sky-bkg level goes around 0 ADU?
Can transform each CCD into the final xy plane by decoding TAN-SIP
(Can conserve fluxes of objects? To What level?) flux conservation should
be better than 5% level in systematic error.
(Can conserve shapes of objects? To What level?)
Can provide mask and variance images, too?
Can stack transformed pixel data by taking median? Yasuda, Lupton
Clipping outlying values available?
Can provide mask and variance images, too?
Can detect any object above 5sigma significance level Lupton
Object parameters to be derived: id, [ x,y, ra, dec (based on TAN-SIP) +
those errors ], flag, [ psfMag + error], [ApertureMag 1",2",3" + errors ],
[petroMag + errors (elliptical apertures if possible), petroRad ], 2nd-order
Can run 104 CCD reduction and calibration on multiple cores based on a
Can collect matchList for astrometry stage onto a node
Can run the astrometry stage by using the match lists
Can re-distribute the result WCS to each node for each CCD
Can detect error events from a pipeline and log any messages from the
To be compared with the original simulated input pixel data
To be tested for number of objects (n(m)), object positions, fluxes, and
Making push all the necessary latest sources to repository and Tag them
Status Comment AI
HD i=25 Should go 1-2 mag deeper in the next challenges
HD Should add mag >22
HD Orientation for gravity should be correctly included in the next stage
HD No pointing offset is tested
HD Corner 4 CCDs should be also handled
HD Resultant flat files should be written into the proper directory
HD Implemented but not yet tested with real CR's
HD Only mechanism is done. In each analysis stage, algorithms to propagate noises should
D Not fully tested. DC1 is ok.
D ok for DC1
D available by detection & measurements code
HD to be finalized with the tagged version codes
HD 5mags are postponed
good for DC1 frames got failure in matching of 3000 frames
HD to be investigated with the tagged version codes
HD under investigation
D Should update the codes with clean simulated data with clean reference sources
D 1% rms in (PSFmag - refmag) for DR8 data. 3% rms for DR7 r', and 5-6% rms for DR7 I' data. basica
HD Improvement in astrometric accuracy is not verified. Need investigation of how to imp
HD Not yet tested
HD envelopes of bright stars seem to be too much subtracted
D working making it fast
NotYet To be tested
HD Algorithms to propagate variances should be investigated
HD same as above
HD Not applied to stacked images,yet.
HD Currently, the following parameters are available: id, [ x,y, ra, dec (based on TAN-SIP) +
NotYet will work on this from now on
NotYet to be derived soon
ectly included in the next stage
nto the proper directory
ysis stage, algorithms to propagate noises should be investigated
mulated data with clean reference sources
s for DR7 r', and 5-6% rms for DR7 I' data. basically robust. The cause of systematic errors should be understood with good simulated data
is not verified. Need investigation of how to improve astrometric accuracy in mosaicking stage.
oo much subtracted
e available: id, [ x,y, ra, dec (based on TAN-SIP) + those errors ], flag, [ psfMag + error], [ApertureMag 1",2",3" + errors ]
rstood with good simulated data