What is FITS? • FITS = Flexible Image Transport System • Invented ~1980 to transfer Radio Astronomy datasets from one observatory to another • Format now used by most astronomical software • Widely used as data archive format • FITSIO and other libraries developed to read/write FITS efficiently using C, Fortran, Perl, Python, Java… • Always big-endian (so portable) • Format defined to the last bit in primary literature. • Images and binary tables fairly space-efficient. FITS Structure in outline • Any number of Header-Data units (HDUs) • Header has metadata: lines of 80 characters NAME = value /[units] other commentary – A few reserved names, e.g. DATE-OBS, no rules otherwise • Data section can be Image or Table – Image: • 8, 16, 32-bit integers, 32 or 64-bit floats • Up to 999 dimensions (mostly 2 or 3 dimensions) – Table • up to 999 columns, up to 2 billion rows • 8/16/32-bit integers, 32/64-bit floats, strings, logical, 64/128bit complex. • Vector columns (optional variable length vectors) • Null values can be defined What’s wrong with FITS • Flat structure for metadata, single name-space, only 8character names • Blocked in units of 2880 bytes • Length of data unit must be declared in header unit, so cannot stream output • No structure to HDUs, just a simple list • No direct access to HDUs, have to decode each header in turn to find size of data unit. • Always big-endian, requires byte-swap on most popular platforms. • Not quite fully 64-bit clean What’s wrong with VOTable • Three alternative ways of including binary: – Pure XML but very verbose, 4-byte real may turn into ~24 bytes, e.g. <TD>-0.12345678E-04</TD> – Blob encoded in Base64 – Pointer to separate FITS file • Possible loss of linkage • Duplication of header data • Tabular datasets only, no format yet for images, spectra, etc. • Cannot use direct offset to required row or field What we want • Flexible header to contain metadata – XML very suitable • Data section to contain heterogeneous data (table) or homogeneous (image etc) but to be efficient, evenly spaced – Binary formats as in FITS very suitable – Endian-neutral – Suitable for streaming data VOTable++ or FITS++? • Use MIME/DIME to have separate sections of file, as proposed for BinX, and have: – XML-based header, very similar to VOTable – Binary data units to follow, similar to FITS • Use BinX/DFDL in header to describe data unit. • Can choose big or little-endian when file created • Allow streaming format for tables, no pre-defined length. • Allow FITS HDU as special case of data unit for compatibility. • Make fully 64-bit clean for tables >2Grows.