AMRC Meteorological Data
Distribution System Part I:
Data Sources and Archive
K. Elena Willmot, Matthew A. Lazzara, Linda M. Keller,
Nicole M. Schroeder
Antarctic Meteorological Research Center
Space Science and Engineering Center
University of Wisconsin-Madison
Introduction
o AMRC since 1992...
o Part I Outline:
o Data Sources
Internal & External
o Data Formats
o Data Archive
Historical & Present
o Distribution
o Challenges
No more FTP?
Access and quality of data
Lack of acknowledgements
o Statistics
(Photo by Michael Forster Rothbart, UW-Madison University Communications)
Data Sources: Internal
o AWS (Automatic Weather Station) data
o Significant portion of internal data comes from UW’s AWS network
o Real-time and archive quality data sets
o Ten minute intervals
o Hourly intervals
o Three-hourly intervals
o 60+ weather stations across the continent
Data Sources: Internal
o Antarctic Composites
o Conceived by UW’s Professor Charles R. Stearns in October 1992.
o Cover South Pole region to about 40° South.
o Satellites used:
GOES, GMS, MTSAT, METEOSAT, FY-2, KALPANA,
NOAA, DMSP, Aqua & Terra, and MetOp
o Created every 3 hours/1 hour every day
o Currently, 4 sizes:
o
o
o
o
512 x 512
683 x 683
1024 x 1024 (standard)
2048 x 2048
o 6 types:
o
o
o
o
o
o
Infrared
Water Vapor
Experimental Visible
Long-wave (Coming soon!)
Short-wave (Coming soon!)
Pseudo Color 
Making Antarctic
Composite Satellite
Imagery:
Data Sources: Internal
o University of Wisconsin’s Space Science and Engineering Center
(SSEC) Data Center
o
o
o
o
Surface synoptic observations
METARs (Meteorological Aviation Reports)
RAOB (Radiosonde Observations)
Ship/buoy observations 
o Numerical Models from GFS
(Global Forecast System) and ECMWF
(European Centre for Medium-Range
Weather Forecasts).
o Polar-orbiting LAC (Local Area Coverage) observations from NOAA
satellites
Data Sources: External
o Participation from various researchers, USAP (United States
Antarctic Program) participants, and international groups within the
Antarctic meteorological community
o International AWS (real-time only)
o USAP research vessel observations
o Field experiment radiosonde observations
o Surface observations at:
o
o
o
o
Palmer Station
McMurdo Station
South Pole Station
Various other field camps
Lawrence M. Gould. Photo by: Al Hickey
http://photolibrary.usap.gov/
NOAA
GAC/LAC
MTSAT
SSEC Data
Center
Data Flow
Meteosat (2)
Products:
GOES (2)
FY-2C
Aqua/Terra
NASA/NOAA
Kalpana-1
AMRC Office
Wisconsin
AWS
International
AWS
USAP Research
Vessels
McMurdo, Palmer
& South Pole
Stations
NOAA
HRPT
DMSP
RTD
Aqua/Terra DB
(McMurdo only)
Data Formats
Archive:
Distribution:
o McIDAS
o McIDAS
o AREA
o MD
o GRID
o AREA
o MD
o GRID
o ASCII Text
o GIF, JPEG
o ASCII Text
o netCDF
o GIF, JPEG
Data Archive: Historical
o Since 1992...
o Media used to archive data:
1.
2.
3.
4.
5.
6.
7.
3
1
Magneto-optic disks
Compact disks (CD)
Digital video disks (DVD)
8 millimeter Exabyte tapes
DDS3 tapes (4 mm tapes)
3590 IBM tapes
LTO tapes
2
6
4
5
7
Data Archive: Present
o RAID/5 disk arrays as well as DVDs are most prominent today
o FTP site most widely used
o Updated monthly with
o
o
o
o
o
AWS – SPAWAR obs
USAP research vessel observations
McMurdo Station obs
Palmer Station obs
South Pole Station obs
o Summary / notification email sent out
Distribution
o Data Services to be discussed in Part II:
o
o
o
o
o
o
o
FTP (File Transmission Protocol)
Rsync Service
Antarctic-IDD (Internet Data Distribution) LDM (Local Data Manager)
ADDE (Abstract Data Distribution Environment)
E-mail
Website
Future:
o RAMADDA (Repository for Archiving, Managing and Accessing Diverse Data)
o GIS (Geographic Information Systems) Software
Challenges:1
o USAP moving away from FTP site – advancing security rules
o Don’t fret! FTP will still be available here at the AMRC:
o Other options: Link with external system
o Limited resources to maintain links
o Manually providing data is expensive and not desired
o We aim to provide data in a more semi-automated, self-service
fashion.
Challenges:2
o Accessing data externally
o Poor quality
o Corrupt files
o Missing
o Gaps in the archive
o Which data can be used / distributed and which cannot?
o Third party redistribution limitations
o Evolving toward nearly fully automated data flow system (from
source to archive)
o Maintenance will be a constant issue – especially at the source
o Documentation
Challenges:3 - Metadata
o Ever increasing need for Metadata
o Significant effort to research/document and provide
o Incomplete historical records, leave some metadata information
missing…
o E.g. South Pole...
 We have found an issue with the South Pole data:
 From 1996 and earlier, the South Pole LCDs (Local Climate
Data) used data that switched between using local time and
GMT time when recording observation times.
 South Pole is working on re-running past LCDs and we do not
know how long it will take / when they will be completed.
Acknowledgements
The authors appreciate the support of the Automatic
Weather Station Program and/or Antarctic Meteorological
Research Center for the data set, data display, information,
etc. ([AMRC/AWS staff member, as appropriate], NSF
grants number ANT-0636873, and/or ANT-0838834)
http://www.art-farm.net/homeGabe.htm
Sculptor Gabriel Warren used a
transparency of the B-15A Iceberg in one of
his pieces called “Interrogation: Dialogue” in
his “Rhegmalogia” collection.
AMRC/AWS Statistics
Data Transferred (Gb)
Web Access (May, 2009 - Apr, 2010)
100
90
80
70
60
50
40
30
20
10
0
May Jun
o Show Gigabytes transferred / month
on Website.
o Average of 70-80 Gb year-round.
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr
o Gigabytes transferred / month on FTP site.
o Generally very variable – maybe due to field
season.
Data Transferred (Gb)
FTP Access (May, 2009 - Apr, 2010)
140
120
100
80
60
40
20
0
May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr
Please join our mailing lists!:
•
•
•
•
•
•
•
ANTIDD – Antarctic-IDD mailing list
ANTDISCUSSION – Antarctic meteorology
discussion group
ANTAWS - Information and updates on
AWS data
ANTOBS - Information and updates on all
other observational datasets
ANTOPS - Notices on AMRC’s real-time
data availability, distribution and status
ANTICEBERG - Notices on latest Iceberg
monitoring images posted to the web site
ANTCOMMUNITY - Infrequent notices to
the Antarctic community on events and
significant developments
How to subscribe:
Visit our new website and click on the “Subscribe” button
next to the mailing list you wish to join:
THANK YOU!
Questions?
Acknowledgements:
National Science Foundation Office of Polar Programs Grant #0838834
South Pole Meteorology Office/Raytheon Polar Services
SPAWAR Office of Polar Programs
McMurdo Station Research Associate
Palmer Station Research Associate
USAP Research Vessels – LM Gould & NB Palmer
McMurdo Ground Station
International partners
& More