Infrared Data Reduction K. Michael Merrill 7/1/2016 1 Windows on the Universe 7/1/2016 2 Electromagnetic Spectrum 7/1/2016 3 Historic Perspective 1960’s - Discrete pixel devices 1968 - Two Micron Sky Survey to K=3 1970’s - AFCRL Rocket Survey 1980’s - IR arrays deployed 1983 - IRAS deployed 1990’s - Rapid growth in array technology 1998 - 2MASS to K=14 2003 - SIRTF 201? - JWST 7/1/2016 4 Atmospheric Transmission 1 - 6 mm 7/1/2016 5 Atmospheric Transmission 6 - 30 mm 7/1/2016 6 Background sky radiation 7/1/2016 7 Background sky radiation OH airglow 7/1/2016 Thermal emission 8 The Operations Challenge in the IR •The sky is always bright (and variable on many time scales) •Site selection •Adopting an observing strategy with active sky subtraction •The telescope can be seen in thermal emission •Reduce mirror emissivity & do not warm baffle •Re-image the telescope mirror inside the instrument & cold baffle •The instrument can be seen in thermal emission •Cool the instrument in vacuum (at or below 77K) •The array can see itself •Cool the array as needed (77K, 30K, 8K, depending on device) •Observations tend to be background (rather than detector) limited •Detecting 2X fainter takes 4X longer 7/1/2016 9 Planck function: black-body radiation Wein’s Law: max 2898mK Emittance = T4 7/1/2016 10 InSb Array Development at NOAO March of the pixels: •58X62 (smallest box) •256X256 •1024X1024 ALADDIN - deployed worldwide •2048X2048 Orion - active development NEWFIRM footprint with 4 Orion detector focal plane mosaic Science in the raw: •H2 gas emission (left insert) •PAH dust emission (middle insert) •JHK color composite (right insert) 7/1/2016 11 Backdrop: 2MASS JHK view of the Orion Nebula Orion Focal Plane Module Clock and Biases Output Current Mirrors Light Baffles Outputs 1-32 Outputs 33-64 Invar36 Pedestal AlN Motherboard Alignment Locator Detector SCA 7/1/2016 Photo Courtesy RIO12 2X2 Mosaic of Orion Modules: 4098X4098 Build a 4Kx4K Focal Plane from four Orion Modules 7/1/2016 13 MBE HgCdTe Cross Section Silicon Read Out IC 7/1/2016 Incident Photons 15 InSb Array Cross Section 7/1/2016 16 •Non-destructive Readout Pixel Readout •Photo-electrons accumulate until reset •Difference between two reads minimizes fixed pattern noise kTC noise Readout Double Correlated Sampling: Fowler 1 0V Time Reset CDS Signal Diode Bias Voltage Reset 0.5 V Readout •Non-destructive Readout Pixel Readout •Photo-electrons accumulate until reset •Difference between two reads minimizes fixed pattern noise kTC noise Readouts Multiple Correlated Sampling: Fowler N (=4) 0V Time Reset MCS Signal Diode Bias Voltage Reset 0.5 V Readouts Read noise 7/1/2016 19 SQIID Optical Schematic 7/1/2016 20 SQIID: dichoric side 7/1/2016 21 SQIID channels from the camera side 7/1/2016 22 Old SQIID 7/1/2016 23 Mosaic: a grid of spatially offset images 7/1/2016 24 Registered composite images at K(2.2mm), H(1.6mm) & J(1.25mm) 7/1/2016 25 NGC 2024: the Flame Nebula Visible:Red 7/1/2016 IR: JHK26 SQIID JHK composite of the Galactic Center Region: 7X7 dithered spatial grid 7/1/2016 27 Sgr A @ K 7/1/2016 28 Star Speeds Around Milky Way’s Black Hole QuickTi me™ and a YU V420 codec decompressor are needed to see thi s pi ctur e. 7/1/2016 29 Galactic Center IR Composite 7/1/2016 30 7/1/2016 31 Sgr A at 3 to 4 microns 7/1/2016 32 Galactic Center in Brackett Alpha Brackett Gamma and Molecular Hydrogen 7/1/2016 33 Galactic Center at 9/13/21 microns Visible 7/1/2016 Near IR 34 Multi-wavelength astrophysics with SQIID: simultaneous operation of 4 arrays sharing a single FOV through dichroics 7/1/2016 M17: the Omega Nebula 35 Image processing: separating the stars from the debri of gas and dust 7/1/2016 36 NGC 7129: JHK SQIID composite 7/1/2016 37 NGC7538: JHK SQIID composite 7/1/2016 38 W3 IRS1 K L L’ composite 7/1/2016 39 M42: the Orion Nebula 7/1/2016 40 Views of Orion Molecular Cloud 1 7/1/2016 41 7/1/2016 42 Egg Nebula in Polarized Light 7/1/2016 43 High Background Science Imaging at the South Pole •NOAO Abu system on SPIREX •two season demonstration •relentless observing •limited by data flow, not natural background Challenge to excel… 7/1/2016 44 NGC6334 - PAH,L,M’ composite S106 7/1/2016 45 S106 7/1/2016 46 Data cube The spatial grid of long slit spectra can be assembled into a 3D structure, then sliced along the dispersion axis (by wavelength) to yield registered images throughout the spectral range. 7/1/2016 47 S106: infrared spectral imaging Observations at the KPNO 1.3m with the Cryogenic Spectrometer (CRSP) at a (2 pixel) wavelength resolution of 2000 across a single spectral baseline from 2.12 mm to 2.25 mm stepping the slit to map the source simultaneously. Left Panel: molecular H2 lines [red:v=1-0 S(1) at 2.122mm; green:v=10 S(0) at 2.224mm; blue:v=2-1 S(0) at 2.248mm]. These line ratios depend sensitively on excitation (fluorescence or dynamic shock) and density. Right Panel: ionized lines of hydrogen Brackett g m and 3 3 [FeIII] (green: G5- H6 at 2.218mm; blue:3G5-3H4 at 2.242mm). These line ratios depend on excitation, density and temperature. 7/1/2016 48 S106 7/1/2016 H2 Br g 49 SQIID Data Processing Overview •The NOAO SQIID Infrared Camera produces simultaneous images of the same field in the J, H, K, and narrowband L passbands, using individual 512X512 quadrants of ALADDIN InSb arrays. •The observations are generally background (photon statistics) limited. •Typical observing programs include: • taking a few (2-5) exposures on the same target with small offsets (to counter ghosts and bad pixels and improve spatial sampling of the images) •taking many exposures of the same target with a dither pattern of offsets (to build up long exposures) - DEEP •spatial mosaics of dithered pairs of images covering larger regions with limited overlap between images (to build up large images) - WIDE. •These three kinds of observations are distinguished because they require somewhat different data reduction strategies. 7/1/2016 50 SQIID Data Processing Overview •UPSQIID software initiated by K. Michael Merrill in 1988 to provide data reduction tools for exploiting the emergent IR array technology at NOAO. •Aim to provide an analysis cookbook for IR arrays with seasoned recipes and to provide a on-site quick-look capability to enhance observing efficiency. •Near-term goal to provide system-wide IRAF routines for general-purpose IR data reduction and multi-wavelength analysis with immediate application to the SQIID four-color infrared camera. •Software "hooks" useful for multi-wavelength image registration incorporated into the IRAF imcombine, iralign/irmatch and imalign/imcentroid tasks. •Using simple IRAF script procedures, a core set of tasks to perform list-based infrared image processing, including sky subtraction, accurate flat-fielding, bad pixel masking and response linearization were generated to regularize the complex task of IR data reduction: usqdark, usqsky, usqflat, usqcorr, and usqproc. 7/1/2016 51 SQIID Data Processing Overview •These programs reside in the “upsqiid” IRAF package as a set of IRAF cl script procedures designed to facilitate the reduction of SQIID datasets. •They have a number of imperfections in the user interface (especially a large number of irrelevant parameters) and currently do not have help files. •Package has evolved from “sqiid” to “sqiid211”, “abu”, “spabu”, & “upsqiid” to meet changes in array size and in IRAF. •Spectroscopic elements have been incorporated to handle low to moderate spectral resolution imaging spectroscopy (grism and grating): slit is stepped orthogonal to the dispersion axis to build up a spectral data cube. •Although not a formally supported IRAF package, it is in wide distribution. •Current location of SQIID information, manuals, and IRAF packages: <http://www.noao.edu/kpno/sqiid/> 7/1/2016 52 SQIID Data Processing Overview •Users are cautioned that IR image datasets often present a greater data reduction challenge than optical CCD images both due to the superior performance of optical CCD detectors (lower dark current, readout noise, and pixel to pixel sensitivity variations) and especially due to the extreme background limited nature of most IR observations. •The results at each step in the process should be carefully examined and problems understood before proceeding. •Many problems can be solved by excluding bad images from the datasets. 7/1/2016 53 SQIID Data Processing Overview •IR observations generally produce spatially composite images. •Since imaging is background limited with integration times measured in minutes, many on-target observations, interspersed with off-target sky frames are required to attain high sensitivity. •Owing to the restricted field of view of IR detectors, sources larger than a few arc minutes are best mapped using a series of pointed observations in an overlapping grid. •Dithering compensates for bad-pixels in the on frames and assists source rejection in the off-frames. •Image overlap assures proper spatial registration and background intensity matching among the individual frames. 7/1/2016 54 SQIID Data Processing Overview •Image registration highly interactive task not generally amenable to automation. •Interrelated scripts produce a generalized registration database used by the IRAF imcombine task to produce a composite image of an arbitrary set of connected images not necessarily sharing a common overlap region. •Spatial registration, using the interactive IRAF centroid task for an overlapping grid and the interactive xyget script for groups of overlapping images, relies on position matching individual stars appearing on multiple images. •Groups of images merged into a registration database by mergecom script. •Since SQIID images at JHKL are taken simultaneously, registration using one channel (usually K) implicitly registers the other channels. • xyadopt script produces registration databases for the other channels from the master channel registration database using the relative linear geometric transformation (offset, rotation, & magnification) between channels based on observations of globular clusters by the interactive getmap script. 7/1/2016 55 SQIID Data Processing Overview •Intensity offsets for data which do not contain a single region in common are determined by the zget script. •nircombine script produces a composite image from the registration database. •Each frame is masked for bad pixels, geometrically transformed to lie atop the master channel frame, shifted linearly in X and Y, intensity offset as required according to the master database and combined into a composite image using the IRAF task imcombine with threshholding (to exclude bad pixels) and median filtering at each pixel in the resultant image. • Careful comparison of photometry of individual and composite images has determined that these modest linear geometric transformations have no significant effect on aperture photometry. 7/1/2016 56 SQIID Data Processing •A "dataset" is the set of direct observations of a given field. •These may be dithered observations of a single target (DEEP) or a mosaic of a larger region (WIDE). • In the extreme case, the dataset may contain only a single exposure (and only the first two steps listed below would be required). •The basic path for the reduction of SQIID dataset can be described as follows: •Create DARK, FLAT, and SKY frames for each passband. •Process each individual source frame to remove instrumental dark current, sky background, and pixel to pixel sensitivity variations. •Combine all images of a given target into a "database". •Interactively define the relative spatial offsets between each image in the database for one of the channels. •Combine the images in the database into a single image suitable for analysis, using bad-pixel masks to exclude bad pixels. 7/1/2016 57 SQIID Data Processing Overview Calculate average background radiation and subtract it from the raw images. Flat-field difference images. Register the flat-fielded difference images. Median through the registered images to produce the final image. 7/1/2016 58 DARK, FLAT, and SKY Frames • Proper data reduction requires accurate solutions for : • the small additive effects of internal illumination and charge generation (DARK frames) • the large additive effects of sky illumination (SKY frames) • the multiplicative effects of position dependent pixel sensitivity (FLATFIELD frames). • Creation of DARK, FLAT, and SKY calibration frames are the first step in the data reduction process. • DARK frames are simple to obtain and process • FLAT and SKY frames are more difficult to create and are crucial to the quality of the final images. IR observations are extremely background limited and the background in the near-infrared is variable at many temporal and spatial scales. 7/1/2016 59 Darks DARK frames (taken with the internal cold dark slide in place) are stable over a night and are probably stable over an entire observing run. Changes in dark current can accompany changes in the array temperature. Since the SQIID dark current has both a base level and a time-dependent component, a dark frame must be created for each exposure time (that you intend to use to determine flat-fields or otherwise require dark subtraction) during the observing run. Composite DARK frames are created with the task “usqdark” : • typically 7 - 9 frames are sufficient • submitted frames are averaged using per pixel minmax rejection • no image scaling or zero-point adjustments are allowed 7/1/2016 60 Flat-Field •Pixel to pixel sensitivity (i.e., the flat-field) is generally stable over the night and perhaps several nights. •Unlike optical CCD observations, images of the interior of the dome or the twilight sky are not practical for the flat-fielding of infrared images. • Since FOV of infrared array detectors now larger than the sources of interest, the sky background may be relatively easier to compensate, but since IR sources are often much fainter than the sky background very precise flat-field calibration is required. • IR background results in sky levels with S/N of better than 100 in single exposures (order 30 sec at J, H, K and order 1 sec at L band). •Taking the median of a reasonable number of blank fields (or target fields without large objects) obtained at different times during the night and at different locations on the sky, a satisfactory measure of the system flat-field can be derived. 7/1/2016 61 Flat-field •Since direct illumination of the array is possible (IR secondary mirrors are usually undersized), observations near bright sources (e.g. the Moon), with atypical illumination should not be used to determine global flatfields. •Observations of illuminated targets inside the dome and during twilight will also have illumination atypical of nighttime observations. Also15 minutes into astronomical twilight, it is doubling every five minutes and you need to be cautious about observing it when it's too bright •Flat frames are examined and compared to those created on other nights. •A common problem is the presence of a large source (e.g. a galaxy) in the center of many observations. This can be ameliorated by the observation of sky frames, the combination of distinctly different observing programs during the same night, and the use of sufficiently large dithering motions. •Post-observation options include the use of carefully selected observations which avoid overlap of the stronger sources. 7/1/2016 62 Flat-Field •The SQIID system response is stable and is very flat across the arrays. •Consequently, the flat-field for each channel should be stable at the percent level under normal illumination and global flat-fields can be constructed which are viable for extended periods of time. •A list of suitable observations with comparable median statistics for each frame, carefully edited to exclude fields with sources larger than the dithering range, is prepared. •Composite FLAT frames are created with the task "usqflat” by: •subtracting a DARK frame (taken with the internal cold dark slide in place) from appropriately selected sky frames •normalize each dark-subtracted frame by dividing by its the median value within the central region •no additional image scaling or zero-point adjustment is allowed •per pixel median filter the ensemble to produce FLATFIELD with high S/N. 7/1/2016 63 Sky •Background illumination in the near IR is far from stable with time or location. •In the near IR (especially at J) moonlight (direct or reflected off clouds) is a factor, and the night sky emission (especially at H, K, and L) is a function of temperature and humidity. H can vary by a factor of 2 and J can vary by 40% on hour time scales. •Although 10 to 30 percent background variations do not strongly limit the S/N of observations (except at K and L for large changes in temperature), they greatly complicate both the creation of mosaics of large regions and accurate surface photometry of objects with extents comparable to SQIID's field of view. •Such observing programs, need a sufficient number of exposures (and intermixed sky exposures if necessary) to create a sky frame for that specific observation. •Programs with single or a few observations of many targets, a sky calibration based on the average over the night is the best that can usually be accomplished. 7/1/2016 64 Sky •After examination of the individual images a list of suitable images is prepared. •Composite SKY frames are created with the task "usqsky”: •submitted frames are brought to a common mean/median within a fixed image subsection using per frame zero-point adjustment and pixel median filtered zero-point adjustments •no image scaling is allowed 7/1/2016 65 UPSQPROC: processing with discrete sky frames Each image (except for dark exposures and blank sky frames) needs to be processed prior to combining the collective datasets into dithered images or mosaics. The processing applied is: •Optionally correct the data for non-linearity •Subtract BLANK frame (DARK or SKY to remove bias and dark current) •Normalize (divide) by the FLAT frame 7/1/2016 66 MOVPROC: processing with moving median sky frames •MOVPROC list processes raw image data using sky frames generated for each processed image. •For a given image the running (moving) median from a subset of frames within a selected list distance of each frame (excluding the given frame; no distinction between on and off source frames) is processed to produce a SKY frame. •The number of frames which enter into the sky can be selected and whether they precede or span the image. •Appropriate for "quick look" type processing of data where the stability of the sky is either unknown or is known to be varying on a time scale shorter than the full data set (grouping skys over long periods is inadequate). •To facilitate list processing, the data processing can begin and end at selected element in the list. •Data which are off source can be optionally included in the running sky frame generation but withheld from full processing into final images. 7/1/2016 67 Average Background Radiation (MOVPROC) A Off-Source Median 7/1/2016 B B On-Source A Off-Source Median 68 PATPROC: processing data in fixed patterns PATPROC processes raw image data generated for representative data acquisition protocols which intersperse object (+) and sky (-) frame in known fixed repeating patterns. These patterns vary in the amount of time devoted to off target images: all_on: + + + + + + all time on source field pair: +- +-+ equal time on and off triad: +-+ +-+ +-+ 2/3 on and 1/3 off alt-triad: -++ -++ -++ quad: +--+ +--+ +--+ equal time on & off with less telescope motion alt-quad: -++- -++- -++- 7/1/2016 69 PATPROC: processing data in fixed patterns •Data are normally processed using the same protocol as UPSQPROC. •Sky frames are generated from the data (sub)set as part of the data processing. •Sky frames can come from all the data, all the off-source frames, or the off-source frame nearest in time to the data sequence to a given frame. •The data at each +|- pattern position can either be single frames or same-size groups of images. •When observing conditions permit one to benefit from a composite sky (improved noise and spatial structure rejection), one can generate composite off-source frames for subtraction from the appropriate on-sky frames. •Each group (size="multiple") off-field images is used to generate composite sky frames for each member of the (size = "multiple") 0 group. The member of a given group of on-source frames is processed using the group sky frame nearest in time in the data sequence. 7/1/2016 70 Average Background Radiation (PATPROC) A Off-Source Median 7/1/2016 B B On-Source A Off-Source Median 71 Building a Database from a Dataset • • • After processing the individual frame, each dataset needs to be converted into a "database". As with datasets, each database contains the observations of a single field or target in a single filter passband. The database construct is used to bring the individual exposures together into the desired final image (dither or mosaic) with appropriate spatial and intensity offset registration. 7/1/2016 72 Building a Database from a Dataset The 'UPSQIID' package contains a set of image registration tools designed to facilitate image registration and combination. The image registration process naturally separates into the following steps: • Determine the relative spatial offsets between images for a selected channel • Link these relative offsets into a single map which contains the offset of each image relative to the origin of the final image. • Determine the relative intensity offsets between images which overlap. • Combine the images into a single image using bad-pixel masks to eliminate bad pixels. • If possible, geometrically transform images at other channels to the selected channel so that they do not need to be spatially registered separately (this is a feature of SQIID). The ’UPSQIID' package image registration tools are designed to operate from ordered image lists. 7/1/2016 73 Building a Database from a Dataset •The initial data flow through the system is determined to a large extent by the degree of spatial overlap among the images in a given data set. •IR observations of extended regions (FOV greater than array FOV) are built up fro spatial grids of pointed data sets which overlap on the edges. •While the degree of spatial overlap is an uneasy compromise between the competing desire to map efficiently and the need to have objects in common along the edges, these data do not in general share a common overlap region. •IR observations of high sensitivity are built up from pointed data sets which are heavily overlapped and share a (generally large) region in common. (The process of producing such data multiply overlapped with relatively small spatial offsets between frames is commonly called "dithering".) •Actual data collection may involve a complex hierarchy spatial offsets (e.g., grids of dithered data) and might require combination of registration techniques. 7/1/2016 74 Building a Database from a Dataset The dataflow for purely WIDE or DEEP observations through the 'UPSQIID' package image registration tools proceeds as follows after generation of the appropriate image list: tile spatially ordered image interactive spatial registration link registration intensity match make composite image 7/1/2016 WIDE usqmos center xylap DEEP xyget zget nircombine 75 Building a Database from a Dataset •These tools generate a database which links the untouched processed data (generally either stored as mosaic of tiled images using USQMOS, or as lists of images) via a prescription which describes how each image is the be transformed, shifted, intensity offset, and masked to make a composite image. •The input images (in the database mosaics or lists) are not themselves modified during this process; rather, each tool in turn inserts the necessary information into the database text files. •When required, temporary copies of the images are created, modified, and discarded at task completion. •One can edit the prescription as required to meet the special needs of each data set. •Tools are provided for merging registration data which share a common image into larger databases. •Ultimately, the images are combined according to the database prescription using the powerful IRAF task 'imcombine' 7/1/2016 76 Registration – Manual Image0nn.fits Image001.fits Compare Click on reference star Click on the same star in every image. Shifts 7/1/2016 77 Registration – Auto Reference Segment Stars Compare Star Positions Shifts Image # n Segment Stars 7/1/2016 78 Combination (Co-adding) Registered difference images Final Image Median through 7/1/2016 79 Correction for system drift and distortions Corrections must be made for: •Intensity non-linearity owing to device physics •Intensity zero-point drifts owing to device physics - now possible using detector reference pixels •Spatial distortion owing to instrument and telescope optics and relative array placement 7/1/2016 80 Cdet (fF) Intensity non-linearity 45 40 35 30 25 20 15 10 5 0 -0.60 Cdet(fF) vs. Vdet(volts) -0.50 -0.40 -0.30 -0.20 -0.10 0.00 Vde t (volts ) Sample InSb detector junction capacitance versus detector bias from 0.0 to -0.6 volt reverse bias 7/1/2016 81 SQIID Linearity Relative error for H (dashed), K (solid), and J (dotted) Channels. Conversion gain is approximately 11 electrons per ADU. 7/1/2016 82 Reference pixels Spatial averaging along a row using reference column to correct for bias drift 7/1/2016 Combined temporal averaging using multiple reads of reference 83 column 7/1/2016 84 7/1/2016 85 7/1/2016 86 7/1/2016 87 Data Pipelining with Abu/Spirex Status Report 3/22/99 7/1/2016 88 The Pipeline is Up and Running Automatic packaging and shipment of files Single point of contact at RIT Automatic logging of sky brightness Automatic logging of standard calibration Automatic reduction and picture assembly – Standard scripts for sky subtraction & dither Based on NOAO abu/upsqiid package 7/1/2016 89 System Performance Routine monitoring of: – sky brightness – sky dips – Photometric reproducibility Working presently only at 3.3um – To baseline performance vs temperature – To establish & debug procedures & protocols 7/1/2016 90 Sky brightness and Ambient Temperature 500 300 Sky Brightness 280 31Jan 300 9Feb 260 10Feb 200 7Mar 100 240 Temperature (Kelvins) Sky (Counts per Second) 400 18Mar Ambient Temperature 220 0 0 7/1/2016 20 40 60 Frame Number (Time Ordered) 80 100 91 Sky Brightness and Stability Sky Brightness & Stability are dramatically different on clear and partly cloudy days – A robust measure of quality – For the telescope operator & for the data user – The pipeline monitors sky brightness for every frame received. – Sky Brightness and Stability will be used by the pipeline as quality metrics 7/1/2016 92 T rying to Observe When Partly Cloudy 1400 Cloudy 9/2 Clear 10/2 Counts per Second 1200 1000 800 600 400 200 0 20 40 60 80 100 120 140 160 Frame Number (30 Second Exposures) 7/1/2016 93 Sky Dip Measurements Measure the sky brightness vs airmass Compare with cold load (LN2 target) Measurements show: – Sky is very dark – Emission from telescope dominant at A.M. < 2 • System emissivity is of order 15% – Uncertainty in “ambient load” dominates calculation – Baffle tube issue is resolved (LN2 signal small) 7/1/2016 94 Skydip Measurements -- 18 March 99 Ambient T emperature = 218 K 500 Counts per Second 400 300 200 Sky Emission (~ 45 Counts/Airmass) Telescope Emission (~100 counts/sec) 100 LN2 target (20 counts/sec) 0 0 2 4 6 8 10 12 Airmass 7/1/2016 95 As Night Falls... Emission from both sky and telescope are observed to decrease No indication of “stray heat” problems Further gains still expected as night falls Sky Dips at 242K and 216K 700 600 skydip@242K skydip@216K 500 Counts per Second 400 300 200 100 0 0 1 2 3 4 5 6 Airmass 7/1/2016 96 Photometric Stability Ice buildup on mirror identified last season as a limiting factor Observing protocol now to point away from incoming wind Photometric stability is improved Repeated Photometric Measurements T wo bright standard stars; delta airmass = 0.34 (over a period of hours) 14.4 Photometric Zero Point (Magnitudes) 14.3 HR2015 HR2015 14.2 14.1 14 HR2451 13.9 0 5 10 15 20 25 30 35 40 Single Short Exposures 7/1/2016 97 7/1/2016 98 The IR Challenge Within the last quarter century, infrared detectors have evolved from individual discrete devices to high tech aggregates of millions of pixels. The scientific drivers for yet more pixels have kept apace - already several projects are dependent on multiple 2K X 2K arrays to reach their goals and next generation facilities envision focal planes paved with detector tiles. To service such focal plane composites requires sophisticated control of multiple devices, but management of the digitized data flow off the focal plane through the data pipeline to the investigator and the archive looms equally large. 7/1/2016 99 Science/Technology Drivers Astronomy continues to move in the direction of larger telescopes, higher spatial/spectral resolution instrumentation, and larger image fields: – Science of scale requires measurement of large areas of the sky at depth – Science of change requires commensurate measurements over time – Indentification and census of rare objects requires mining of large areas – Need for sample completeness and statistical accuracy requires measurement of multiple sources at the same time Scarcity & high cost of observing resources demands an observing environment that: – reliably delivers accurately calibrated, repeatable observations – must be flexible - optimized to meet the diverse needs of the science – must be adaptable to accommodate changing needs – is capable of relentless operation with high efficiency Science data flow requires: – uniformity in acquiring, processing, and archiving image data – rapid turnaround of very large data volumes – optimal coupling with data processing, instrument, and telescope systems 7/1/2016 100