Project CRAFT: A Test Bed
for Demonstrating the Real
Time Acquisition and Acrhival
of WSR-88D Base (Level II) Data
Presented to UCAR/Unidata Program
12 October 2000
ns f
Kelvin K. Droegemeier
Center for Analysis and Prediction of Storms
and
School of Meteorology
University of Oklahoma
NEXRAD Doppler Radar Network
NEXRAD Facts and Figures

158 radars (141 in the Continental US)
– 120 National Weather Service radars
– 26 Department of Defense radars
– 12 Federal Aviation Administration radars
WSR-88D Components
Radar Data
Acquisition
ns f
WSR-88D Components
Radar Product
Generator
ns f
WSR-88D Components
Principal User
Processor
ns f
WSR-88D Components
Principal User
Processor
ns f
NEXRAD Data Types




Archive Level I (analog, time domain output)
Level II data (digital base data at full resolution)
Archive Level III (“NIDS” digital products from
the Radar Product Generator)
Archive Level IV (at the PUP)
NEXRAD Data Types




Archive Level I (analog, time domain output)
Level II data (digital base data at full resolution)
Archive Level III (“NIDS” digital products from
the Radar Product Generator)
Archive Level IV (at the PUP)
NEXRAD Level III (NIDS) Data



24 products available from all CONUS radars
in real time
Provided by 3 commercial vendors
Lowest 4 elevation angles of the radial
velocity and reflectivity
– 1 degree x 1 km resolution
– Values are quantized (e.g., 0-5, 5-10, 10-15)
ns f
ns f
NEXRAD Data Types




Archive Level I (analog, time domain output)
Level II data (digital base data at full resolution)
Archive Level III (“NIDS” digital products from
the Radar Product Generator)
Archive Level IV (at the PUP)
NEXRAD Base (Level II) Data

Full resolution digital data
– Full data precision
– All tilts
– Not quality controlled



1 degree x 1 km resolution for reflectivity
1 degree by 0.25 km resolution for radial
wind
Not available in real time except for
selected sites (more on that later)
ns f
Base Data Usage: NSSL Warning
Decision Support System on 3 May 1999
ns f
Courtesy National Severe Storms Laboratory
Trimmed Detections and Ground Truth Damage Paths
Hits (142)
Misses (25)
FAs (21)
Courtesy D. Zittel
5 April 1999 - Impact of Base Data
Initial 700 mb Vertical
Velocity Using NIDS
Initial 700 mb Vertical
Velocity Using Level II
Data and SDVR
12 Z Reflectivity
ns f
Courtesy S. Weygandt and J. Levit
5 April 1999 - Impact of Base Data
15 Z Reflectivity
3 hr ARPS CREF
Forecast (9 km) Using
NIDS Data
Valid 15Z
3 hr ARPS CREF Forecast
(9 km) Using Level II
Data and SDVR
Valid 15Z
ns f
Courtesy S. Weygandt and J. Levit
Recording Level III WSR-88D
Data

Limited set of products recorded on CDROM at the radar site
ns f
WSR-88D Level III Data Archival
at NCDC
Stations
Media
Bytes/media
Daily Receipts
Annual Receipts
120
CD
600 MB
4 GB
60 TB
ns f
WSR-88D Level III Data Processing
NWS
Disk Storage
PUP
Disks
Archive
To NCDC
Record
Disks
PC
Copy Files to HDSS
Disk Drive
HDSS
Courtesy Steve DelGreco (NCDC)
Level III Requests at NCDC
ns f
Recording WSR-88D Level II
Data


Not in original plans
Exabyte 8 mm tape recorders added to all
radars (1994) as an interim solution
ns f
WSR-88D Level II Data Archival
at NCDC
Stations
Media
Bytes/media
Daily Receipts
Annual Receipts
158
8 mm tape*
4.8 GB
165 GB
60 TB
*6-step process with NCDC operating 92 Exabyte tape
drives
ns f
WSR-88D Level II Data Processing
Courtesy Steve DelGreco (NCDC)
Level II Requests at NCDC
ns f
WSR-88D Data Archival at NCDC
% Rcpts
#Sites
Level II NWS
Level II DoD
Level II FAA
64.9
35.7
9.5
120
26
12
Total 88D Network
55.7
158
Level III NWS
89.5
120
ns f
WSR-88D Base Data Archival at
NCDC








Currently 42,000 8-mm tapes in archive
9,500+ copies distributed to date
Cost per tape is nominal ($100 for a radar day) but
high for large (multi-year) data sets
Major time delays (2-4 months; human resources)
Archive incomplete
No mechanism for perusing the data base
NOAA spent $768,000 on Level 2 recording system
repairs from 6/1/98-6/14/99
NCDC maintains 92 Exabyte tape systems
ns f
Summary: WSR-88D Radar Data


The scientific and operational communities need
base data (real time and archived)
Although NIDS data are available in real time from
all WSR-88D radars, they are insufficient for many
applications (NWP, hydrology)
– Degradation of precision
– Only the lowest 4 tilts are transmitted

Base data currently are not available in real time
– Originally would have been expensive
– Presumed large volume of data (10 mbytes/5 min/radar)
– Need wasn’t there 10 years ago

The technology and need now exist to prototype
the direct acquisition, use, and archival of base
data in real time
ns f
The Collaborative Radar
Acquisition Field Test (CRAFT)

Establish a prototype real time WSR-88D base data
acquisition test bed to
– Evaluate strategies for compressing and transmitting
base data in real time
– Develop efficient and cost-effective strategies for direct
digital ingest, archive, and retrieval at NCDC
– Assess the value of base data in numerical weather
prediction
– Test web-based data mining techniques for rapid
perusal/access of base data by the scientific community
ns f
Participants

CRAFT was established by the OU Center for
Analysis and Prediction of Storms (CAPS) in
collaboration with the
– UCAR Unidata Program Office
– NOAA WSR-88D Operational Support Facility
– University of Washington

CAPS and the NOAA National Severe Storms
Laboratory are now jointly coordinating the effort
in conjunction with the NEXRAD OSF and UCAR
ns f
RIDDS: The Foundation of CRAFT
ns f
Technical Strategy
At the radar site
Dedicated 56K line
Cisco 1600
Series
Router($2000)
($2000 - $6000/year)
Server
WSR-88D
Internet
RIDDS
Linux PC
Unidata LDM
($1500)
Users
Repeater Hub
ns f
Original CRAFT Network
ns f
Sharing Data from Multiple
Sources via Cooperating LDMs
Source
LDM
LDM
LDM
Source
Source
LDM
LDM
LDM
Internet
LDM
LDM
LDM
ns f
Courtesy D. Fulker, Unidata
Technical Highlights
– We added to LDM in-line
compression/decompression (BZIP2 off-the- shelf
algorithm)



maximum 16:1 in clear air
about 8:1 in the worst case
average of about 12:1
– Maximum aggregate data rate is 15
Mbytes/radar/5 min uncompressed = 48
Mbits/sec compressed for entire national network


bandwidth isn’t the issue, but rather topology, quality of
service, and data storage
one year’s worth of compressed data from NWS radars
is about 10 petabytes
ns f
Technical Highlights
– Compression now being improved (hope for >
20:1)
– LDM relays data to other nodes


unlimited system growth
redundancy
– Simultaneous data transmission to multiple users
– Single point of failure (modem is backup)
ns f
Real Time Management and Reliability: >95%
ns f
http://traffic.onenet.net/
Real Time Management and Reliability: >98%
Steve Goodman, NASA/MSFC
CRAFT Phase I: Proof-of-Concept
ns f
Internet2 and Abilene

Internet2 is ...
– a project, led by universities, to develop
advanced internet technology and applications
vital to the research and education missions of
higher education
– big commitment


member universities have committed over $70 million
per year in new investment on their own campuses
corporate members have committed more than $30
million over the life of the project
ns f
Internet2 and Abilene

Abilene is ...
– an advanced backbone network testbed




that connects regional gigaPoPs
supports Internet2 applications development
that is focused exclusively on university research and education
that is developing next generation operational and quality of
service capabilities
ns f
Abilene Network
January 1999
Seattle
New York
Sacramento
Denver
Indianapolis
Kansas City
Los Angeles
Atlanta
Abilene Router Node
Abilene Access Node
Operational January 1999
Planned 1999
Houston
New Concept: Abilene/Internet2 +
NEXRAD
ns f
New Concept: Abilene/Internet2 +
NEXRAD
ns f
New Concept: Abilene/Internet2 +
NEXRAD
ns f
New Concept: Abilene/Internet2 +
NEXRAD
U-WA
NCEP
AWC
NCAR/FSL
NCDC
OU
TPC
ns f
Logical Network Topology
Other LDMs
(university, private,
government)
Via Abilene or
Commodity Internet
OU
LDM
FSL
LDM
U-WA
LDM
Abilene Backbone
NCAR
LDM
NCEP
LDM
NCDC
LDM
56K Phone
Lines to Abilene
Universities
ns f
Adapted from D. Fulker, UCAR
Alternative Strategy
At the radar site
Internet Direct
Cisco 1600
Series
Router($2000)
Server
WSR-88D
Internet
RIDDS
Linux PC
Unidata LDM
($1500)
Users
Repeater Hub
ns f
Alternative Network Topology
Via Abilene or
Commodity Internet
OU
LDM
FSL
LDM
U-WA
LDM
Abilene Backbone
NCAR
LDM
Other LDMs
(university, private,
government)
NCEP
LDM
NCDC
LDM
Direct Connection
to the Internet in
the NWS Office
ns f
Adapted from D. Fulker, UCAR
The Benefits
– Configuration completely scalable
– Everyone has real time access to WSR-88D base
data (all radars)





Special quality of service for key users (NCEP, SPC)
Leverages existing infrastructure and serves as a
means for developing advanced networking
capabilities
Easy linkage with other networks (e.g., AWIPS,
NOAAPort)
Could easily add other radars
Significant capacity for future growth (dual-pol,
phased array)
ns f
The Benefits
– Rapid and less expensive base data archival and
retrieval at NCDC






Number of steps reduced to 2
Archive time reduced by factor of 4
Will improve archival rate (now 65%)
Reduces data request turn-around time
Reduces data volume via compression
Eliminates tape drive costs
– Possible secondary back-up/real time archives
where advanced data perusal and web-based
data mining tools will be available
ns f
The Challenges






Operational reliability and redundancy in a
research network
Network security
The future of the I2/Abilene network
infrastructure
Coordination with regional/state networks
Capacity for growth
Phasing with deployment of the Open RPG
ns f
Challenges for Unidata




Use of LDM could expand significantly
How to provide LDM support to private
sector users?
Integration of LDM into NEXRAD Open
Systems Architecture (more later)
End-user visualization and analysis software
(not just a UCAR problem)
ns f
CRAFT-2
One of 4 National Abilene Initiatives
 Expand CRAFT-1 to additional 30-40
radars
 Possibly include FAA radars
 OU and NCAR have an opportunity to
become joint online radar archive sites for
data mining, visualization, etc

ns f
ns f
CRAFT-2 Radars Now Delivering Data
May soon begin turning off
some of the 8 mm recorders!!
ns f
At NCDC Today
Abilene (T1
into NCDC)
WSR-88D
HDSS
LDM
NEXRAD Base Data
from the 12 CRAFT Radars
Mass Storage
ns f
Opportunities
– Web-based data mining
– Analysis and visualization packages
– Networking quality of service research
– Digital libraries and distributed storage
technology research
– Storm- and meso-scale data assimilation
and prediction
– Mesoscale process studies
– Small-scale climatologies
ns f
http://geosciences.ou.edu/~jsnow/Research/Meso/
MS Thesis by Brad Pugh (1999)
Oklahoma Climatological Survey
Funding Status
– NOAA ESDIM Grant funded (CAPS+NSSL+OSF+NCDC)


$540K/3 years
Research Thrusts
– Test of direct ingest/archival at NCDC
– Improve compression algorithms
– Initial work on web-based data mining
– NOAA earmark funding to OU




$474K for 1 year
Expand CRAFT to 30 radars (CRAFT-2)
Develop data assimilation capabilities for the WRF model
Kelvin doing a mini-sabbatical at the NSSL this fall
– HPCC Proposal ($150K for 1 year, about to be funded)



Data mining
Network quality of service research
Hardware for additional radars
ns f
The Future: FAA Radars

Initial efforts are now underway to
explore extending the CRAFT
infrastructure to FAA radars
– TDWR (terminal Doppler weather radar)
– ASR (airport surveillance radar)
– ARSR (air route surveillance radar)
ns f
TDWR
ns f
Weber (2000)
Airport Surveillance Radars (ASR-9)
ns f
Weber (2000)
Airport Surveillance Radars (ASR-11)
ns f
Weber (2000)
Air Route Surveillance Radars (ARSR-4)
ns f
Weber (2000)
Open RPG
– NEXRAD is implementing an open-systems
software standard based on UNIX and other
widely-used technology (e.g., TCP/IP)
– The temporary RIDDS system becomes the Base
Data Distribution System (BDDS)
– Will require moving LDM and compression into
BDDS
ns f
Open RPG
Phase I: Current Strategy
At the radar site
Internet Direct or 56K
Phone Lines
Internet2/
Abilene
WSR-88D
RIDDS
Linux PC
Unidata LDM
($1500)
Users
Repeater Hub
ns f
Phase II: BDDS + LDM/PC
At the radar site
Internet Direct or 56K
Phone Lines
Internet2/
Abilene
WSR-88D
BDDS
Linux PC
Unidata LDM
($1500)
Users
Repeater Hub
ns f
Phase III: Full BDDS
At the radar site
WSR-88D
Internet Direct or 56K
Phone Lines
BDDS +
LDM
Internet2/
Abilene
Users
Repeater Hub
ns f
Summary







Efforts are now underway to create a national WSR88D Level II (base) data ingest system
Real-time access for all users by subscribing to
desired radars via LDM – LDM is a KEY component!
ESDIM project and CRAFT-2 are investigating key
issues (reliability, scalability, security)
Hope to have an online digital library in Norman for
data mining and research with another archive at
NCAR
Pacing item for implementation is Open RPG/BDDS
deployment (scheduled June ’01 – Dec ‘02)
New build for BDDS will be phased in simultaneously
Tremendous opportunities for research
ns f