Intermountain West Data Warehouse
3SAQS Technical Workshop
April 30th, 2015
Data Warehouse
Current and Future Development
Outline
•
•
•
•
•
Update - DW Overview & Development
Data Request Status
Visualization tools
– Image browser link
– Model-to-obs tool
– Emissions review tool
– Dynamic Geospatial Model visualization tool
– ENVIRON Source Apportionment tool ENVIRON site
Functionality
• Data Request Form
• Prototype data selection form
• Category Management
• Inventory Management
• Forum & email notifications
Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015
Outline
•
•
•
•
•
Update - DW Overview & Development
Data Request Status
Visualization tools
– Image browser link
– Model-to-obs tool
– Emissions review tool
– Dynamic Geospatial Model visualization tool
– ENVIRON Source Apportionment tool ENVIRON site
Functionality
• Data Request Form
• Prototype data selection form
• Category Management
• Inventory Management
• Forum & email notifications
Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015
IWDW Overview
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015
IWDW Overview
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015
DW Development Update
• Project Management
– Q1 2015
• Data Requests; Inventory Management; Data Delivery
• Documentation
• Data & Information Management
– Q2 2015 through 2016
• Visualization tool development (modeling I/O & analysis)
• Modeling platform review
• Expanded documentation and “easy-to-reference” information
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015
Outline
•
•
•
•
•
Update - DW Overview & Development
Data Request Status
Visualization tools
– Image browser link
– Model-to-obs tool
– Emissions review tool
– Dynamic Geospatial Model visualization tool
– ENVIRON Source Apportionment tool ENVIRON site
Functionality
• Data Request Form
• Prototype data selection form
• Category Management
• Inventory Management
• Forum & email notifications
Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015
Data Request Summary – pg. 1
Request
Direct Interest Studies for
WSAQS Cooperators
Requestor
Data
Supporting
Docs
Request
Submitted
3-State
Steward
Action
Filled
4/24/2015
Methane/VOC Control Sensitivity
Study
Tammy Thompson
CIRA
2008 Emissions
Scope of Work
Yes
Emissions Inventory Data Summary for
2008
Tammy Thompson
CIRA
2008 Emissions
None
Yes
Mike Barna
Source Apportionment Modeling for
Nitrogen in Greater Yellowstone Area
(GYA)
Tammy Thompson
CIRA
2011 Emissions
(unreleased)
Project Plan
Yes
Mike Barna
National Park AQRV
Todd McDonnell
E&S Environmental
Chemistry
2011 Emissions
(unreleased)
None
Yes
Routes to Sustainability for Natural Gas
Development and Water and Air
Resources in the Rocky Mountain
Region - Sensitivity Analysis
Shannon Capps
University of
Colorado, Boulder
2008b
Base year
Platform
(partial)
Modeling
Protocol and
Timeline
Yes
Comparison of source contributions to
nitrogen deposition at Class 1&2 areas
in the Western U.S.
Tammy Thompson
CIRA
2008b Emissions
and CAMx inputs
None
No
WRAP TSS Emission Tool Updates –
2011a emissions displays
Cassie Archuleta Air
Resource Specialists
2011a Emissions
(unreleased)
None
No
2008b Base year
Platform
None (may be
taken care
under
Archuleta
request above)
Inventory--categories and methodology
consistent with previous WRAP
inventories for regional haze from 2004
and 2008 (reasonable progress)
Daniel Bon CDPHE
Yes
Kevin
Briggs
Filled
4/24/2015
Tom Moore
Tom Moore
Will
fulfill via
TSS after
update
w/2011
emissions
Data Request Summary – pg. 2
Request
Requestor
Data
Supporting
Docs
Request
Submitted
3-State
Steward
Hiawatha Regional Energy
Development Project (HREDP)
Environmental Impact Statement
Ralph Morris
ENVIRON
2008b NEPA
Platform
None
No
Charis Tuers
Converse County Development
Environmental Impact Statement (EIS)
Chao-Jung Chien
AECOM
2008b NEPA
Platform
None
Yes
Ryan
McCammon
Colorado Oil and Gas Sensitivity
Analyses
James Wilkinson
Golder Associates
Inc.
2008b NEPA
Platform
None
Yes
Update of a USFS Fire handbook
Suresh Ratnam
University of
Southern
California
Base2020_08a
None
No
Improving Ozone modeling in Uintah
Basin
Huy Tran Utah
State University
2008b
Emissions and
utilities
None
Yes
incomplete
Analysis of FRAPPÉ and DISCOVERAQ 2014 campaigns
Gabriele Pfister
NCAR
2011a
Emissions and
utilities
(unreleased)
None
Yes
Action
NEPA Studies
Outside
Requests
(applicability to
WSAQS
Cooperators not
clear)
Fulfilled
3/18/2015
Responding to Data Requests
• Process needs for operating the Data
Warehouse noted so far:
– the identities, roles, and responsibilities of
cooperating agency “stewards” need to be
clarified
– the necessary level of documentation and
applicability to the overall goals of the
cooperators; and
– the form of a “release” of a platform and its
component data, including the associated use
recommendation.
Pending Project Proposals with no data requests
Information or Data Request
Proposal
Requestor
Start Date
Data
Request
Submitted
Fulfillment Approach
Emissions QA 2011 Base A
Gail Tonnesen EPA
Region 8
10/1/2014
None
No
Internal project report completed
Ozone spatial plots for 2011 MPE
Gail Tonnesen EPA
Region 8
10/16/2014
None
No
Display capability on DW in
development
PEM integration with
SMOKE/CMAQ
Private consultant
11/11/2014
None
No
Cancelled
2011a Modeling Sensitivities
Zac Adelman UNC-IE
9/1/2014
None
No
Conducted as internal DW
modeling study analysis
PMDETAIL
Ralph Morris ENVIRON
12/1/2014
None
No
Pending, will utilize files from
DW and PMDETAIL project
2020 Future Case Sensitivity
Mike Barna National
Park Service, Air
Resource Division
1/23/2015
None
No
Pending
Action
Outline
•
•
•
•
•
Update - DW Overview & Development
Data Request Status
Visualization tools
– Image browser link
– Model-to-obs tool
– Emissions review tool
– Dynamic Geospatial Model visualization tool
– ENVIRON Source Apportionment tool ENVIRON site
Functionality
• Data Request Form
• Prototype data selection form
• Category Management
• Inventory Management
• Forum & email notifications
Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015
Image Browser
•
http://views.cira.colostate.edu/tsdw/Tools/ImageBrowser.aspx?pathid=TsdwPlotsRoot
• Displays large image sets delivered by data
provider
• Possible Next Steps
– Improve navigation (fewer mouse clicks)
– Display meta data
– Browse multiple image sets side by side
Model-To-Obs Tool
•
http://views.cira.colostate.edu/TSDW/modeling/modelobscomparison_iwdw.aspx
• Compare model values to observed values by
network
• Possible Next Steps
– Platform selection
– Additional filtering capabilities
Emissions Review Tool
• http://views.cira.colostate.edu/tsdw/Emissions/ReviewTool.aspx
• View modeled emissions by source category,
state, county, and SCC
• Possible Next steps
– Refactor data base to correctly label and
aggregate parameters and source categories
– Expand SCC visualization capabilities
Dynamic Geospatial Model Visualization
• http://views.cira.colostate.edu/tsdw_Dev/Maps/ModelDataGIS.aspx
• Displays model output data with overlaid monitor data
Dynamic Geospatial Model Visualization
• ArcSDE (Spatial Database Engine) with SQL Server
– Stores/Serves rasters of model data
– Stores/Serves raster attributes
– Data can be accessed remotely by ArcGIS users
• ArcServer
– Provides web services which render data stored in SDE
– Web Services consumed by web applications such as
the data warehouse
Dynamic Geospatial Model Visualization
• Advantages
– Map Projection
– Uniform color scale
– Monitor data overlay
– Ease of combining multiple features
– Sharing data with other GIS users
Dynamic Geospatial Model Visualization
• Disadvantages
– More Pre-processing
– Data duplication
– Infrastructure heavy (ArcServer, ArcSDE)
Dynamic Geospatial Model Visualization
• Possible Next steps
– Parameterize, platform, species, layer,
time window
– Additional Monitor data
– Additional features
• Cities
• Roads
– Automate preprocessing, execute on
demand
Source Apportionment Tool
• http://ims.environcorp.com/aqms_test
• Developments/Status
• Received prototype from Environ
• Database up and running
• Possible Next steps
• Deploy prototype user interface on CIRA
systems
• Integrate user interface with main
warehouse system
• Integrate database with main warehouse DB
• Automate data ingestion
Outline
•
•
•
•
•
Update - DW Overview & Development
Data Request Status
Visualization tools
– Image browser link
– Model-to-obs tool
– Emissions review tool
– Dynamic Geospatial Model visualization tool
– ENVIRON Source Apportionment tool ENVIRON site
Functionality
• Data Request Form
• Prototype data selection form
• Category Management
• Inventory Management
• Forum & email notifications
Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015
Data Request Form
• Automated FTP fulfillment
– One click FTP user creation and data access
– Create/Update FTP user account
– Enable FTP access to requested data without
duplication of data
– Two Successful deliveries via FTP
– Image Browser uses FTP fulfillment to serve
images
Data Request Form
•
http://views.cira.colostate.edu/tsdw_dev/DataRequest/PlatformSelection.aspx
• Component Selection
– Flattened tree
– More descriptive information
– Alternative to category tree
Data Request Form
• Possible Next Steps
– Integrate component selection with request form
– Possibly separate project proposal from data
request
– Continue to evolve approval and stewardship
policies and procedures
– Integrate data release status and/or evaluation
level
– Notification email clean-up
Category Management
• Create/Remove/Edit
– Data Categories
– File Items in category
– Meta Data
• Browse file system
• Data Management Reports
– Missing data
• Possible Next steps
– Additional reporting as needed
– Better display of category/file meta data parameters
– File/Category history
Inventory Management
• MD5 checksums stored in relational database
• History of file meta data
– file size
– mod time
– checksum
• Data Recovery tool
• Next Steps
– Incorporate detailed meta data scraper
– Automation of Checksum computation
– Reporting
• Report changes to data requestors
• File level data status
Forum & Email Notifications
• http://views.cira.colostate.edu/tsdw/Forum/
Outline
•
•
•
•
•
Update - DW Overview & Development
Data Request Status
Visualization tools
– Image browser link
– Model-to-obs tool
– Emissions review tool
– Dynamic Geospatial Model visualization tool
– ENVIRON Source Apportionment tool ENVIRON site
Functionality
• Data Request Form
• Prototype data selection form
• Category Management
• Inventory Management
• Forum & email notifications
Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO
April 30, 2015