3SAQS Data Warehouse Update
October 10, 2014
CIRA - Fort Collins, CO
Outline
•
•
•
•
•
•
Data Warehouse Team
System Overview
Phase 1 Operational State
Website Overview
Project & Request Workflow
Future Work
3SAQS Steering Committee
October 10, 2014
2
Data Warehouse Team
•
•
•
•
•
Shawn McClure
Tom Moore
Dustin Schmidt
Rodger Ames
Cassie Archuleta (ARS)
3SAQS Steering Committee
October 10, 2014
3
System Overview
Monitoring
Data
Modeling
Platforms
IMPROVE,
STN, Met, etc.
UNC,
ENVIRON,
CIRA, etc.
External modeling
activities
PGM and AQ
impacts
assessment by
consultant
Inputs
ROD/EIS
published
by Agency
Sponsor
Modeling platforms,
monitoring data, completed
modeling studies are
uploaded along with related
documentation.
Project
Management
Delivery
Modeling files for a specific
request are obtained by the
user. Each data request is
accompanied by a manifest.
After receipt, user signs off on
request.
Project Manager/Data User
initiates a project on the DW.
Can assigns collaborators
who can submit a data
request.
Data
Request
3SAQS Steering Committee
October 10, 2014
User submits request within
an active project. Request
may be for pre-defined data
package (current), or a la
carte selection (future).
5
Data Warehouse – Phase 1
• Project Management System
– Project/inventory tracking, modeling platform
request form (Phase 1=NEPA), data manifest
• Hardware Upgrades
– New 55tb RAID
– USB3 transfer to external disks
3SAQS Steering Committee
October 10, 2014
6
Data Warehouse – Phase 1 (cont.)
• Modeling Platforms
– Base 2008b, Base 2011a CAMx (release pending
approval)
• Data deliveries
– Hiawatha (ENVIRON)
– Converse County (AECOM)
• Website
– Pages for studies, projects, documents, procedures
and results, meetings & events, literature, glossary,
about (updates ongoing)
3SAQS Steering Committee
October 10, 2014
7
Data Warehouse – Phase 1 (cont.)
• Inventory Management
– Data is placed within the warehouse
– Warehouse file system is represented in database
• Facilitates change control
– File system items are data
• Files
• Directories
– Categories are groups of File System Items
•
•
•
•
Files and Directories are grouped into Categories
Meta data is attached to Category
Categories may be hierarchically related
Data may belong in multiple categories
– Data Packages are groups of Categories
• Collection of categories for intended use (NEPA)
Website overview
• Studies
– Modeling platforms (completed and forward looking)
– Monitoring data overview (pending)
• Documents
– Modeling platform protocols, MPEs, etc.
• Procedures and Results
– Specific how-to pages for DW web based tools (pending)
• About
– General information about 3SAQS and the DW (pending)
• Projects
– User initiated AQ modeling using 3SAQS platforms
3SAQS Steering Committee
October 10, 2014
9
Demo Website
http://views.cira.colostate.edu/TSDW/Demo/
3SAQS Steering Committee
October 10, 2014
10
Data Warehouse Website – Phase 1
User
Documents
3SAQS Steering Committee
October 10, 2014
Project
Management
Tools
11
3SAQS Steering Committee
October 10, 2014
12
Studies
3SAQS foundational modeling platforms
• Phase 1 (done)
– 2008b available
– 2011a (release pending)
• Phase 2 (forward looking)
– 2008_2020a
– 2011_2020a (CAMx/CMAQ)
– Preliminary 2014 (CAMX/CMAQ)
– 2014_FutureCase (CAMX/CMAQ)
3SAQS Steering Committee
October 10, 2014
13
3SAQS Steering Committee
October 10, 2014
14
Documents
•
Memorialize work that is already done
–
•
Reference basis for modeling platforms that the DW is delivering
–
–
–
•
WestJump to 3SAQS
3SAQS and 2011 NEI O&G Emission Inventories reconciliation
3SAQS Methane inventory & response to comments
Modeling Guidance
–
–
•
•
Base 2008b MPE
2011 WRF MPE
Technical memos
–
–
–
•
2008, 2011 Modeling Protocols & response to comments
Calls for comments on draft documents
–
–
•
Modeling protocols
MPE reports
Emissions reconciliations
Reports and comments
–
•
Network Assessment
NEPA/non-NEPA guidance
SMOKE
Work plans
Annual & Quarterly Reports
3SAQS Steering Committee
October 10, 2014
15
3SAQS Steering Committee
October 10, 2014
16
Procedures and Results
• “How to” pages for:
– Data download and upload procedures
– Application of visualization tools
– Understanding modeling platforms
– Interpreting emissions inventories
– Accomplishments and future work of the 3SAQS
and DW
3SAQS Steering Committee
October 10, 2014
17
About
• Informational Flyer
– Goals of 3SAQS and role of DW
– Studies and project management overview
– Modeling toolbox highlights
• Geospatial data selection
• 3SAQS and cooperators
–
–
–
–
BLM
NPS
Forest Service
States of CO, WY, UT
3SAQS Steering Committee
October 10, 2014
18
Projects
User initiated AQ modeling projects
• NEPA Projects
– Hiawatha (ENVIRON)
– Converse County (AECOM)
– Others pending
• Non-NEPA (future work)
–
–
–
–
–
–
NPS N deposition
NPS Vertical Profiles of simulated ammonia
EPA QA 2011 Base A
State of CO RAQC O3 SIP
State on NM O3 nonattainment
Others…
3SAQS Steering Committee
October 10, 2014
19
Project Workflow - High Level Goals
• Manage Data Access
• Provide Project Managers an overview of data delivery
progress
• Provide Users a concise view of available data
• Provide Data Requestors with a detailed manifest (receipt)
• Detailed data accounting (Inventory Management)
– Meta data
– Who got what and when
– Change control
• Track usage of supplied data
• Capture communication threads (Future work)
Project Workflow - Overview
• 3SDW Administrators designate registered users as “Project
Managers”
– Cooperating Agency point of contact
• Project Manager creates new analysis project
– Provides high level meta data
– Designates project team members
• Contractors, Subordinates, Etc
• Project team member initiates a Data Request
– Data Selection
– Provide detailed meta-data
• 3SDW Team fulfills data request
• Project team member signs-off request as completed
3SAQS Steering Committee
October 10, 2014
21
Project Workflow – User Roles
Project Workflow - User Roles
• Registered User
– View projects (Read-only)
– View Data Requests (Read-only)
Project Workflow - User Roles
• Project Manager
– A Registered user who is also a member of a
3SDW cooperating agency (BLM, EPA, NPS, etc).
– Create new analysis projects
– Assign project team members
– Monitor data transfer progress
– Issue data requests
Project Workflow - User Roles
• Project Team Member
– Assigned to team by Project Manager
– Issue data requests for assigned projects
Project Workflow
Projects Overview
• Purpose
– Present an overview of Data Request status
– Entry point for Project process
– Create a new project
Projects Overview
Projects Overview
Project Configuration
• Purpose
– Capture high level information about the project
• High level description (Details provided by data
requestors)
• Location
• Timeframe
– Managing Data Access
• Selected pool of Project Managers (3SDW Cooperators)
• Project Managers assign 3SDW registered users to
project
Project Configuration
Project Configuration
Data Request
• Allow users to browse available data
• Capture optional information about
– Intended use of data
– Results/Conclusions from data usage
• Supply detailed manifest of requested data
Data Request
Data Request
Data Request
Data Request
Future Work
•
•
•
•
•
Provide finer granularity in data selection
Tune process for supplying supplemental data
Web interface for Data Cataloging & QA
Enhance performance of online data transfer
Capture e-mail communication traffic for
Project / Data Request
• Data Submission Process (Return model
results to warehouse)
Data Warehouse Website – Phase 2
Documents
Sign in/
Register/user’s
account information
> Software benchmarking
> Modeling platforms
> Modeling Protocol
> MPE reports
> Glossary
> Modeling and website
guidance
Contact us
Site admin
User
MPE plot browser
News and Events
Documents
> Development milestones
> Links
Project
Management
Tools
Emissions review
tool
Literature
O&G construction
and production
emissions Excel
work books
> Articles on air quality
modeling related to O&G
developments in the
western region
Studies
> High level view of
foundational modeling
platforms either available
or pending in the DW
Projects Overview
> status of projects
> project access can be
restricted to specific
users/group
Project Configuration
> create new projects
> edit existing projects
> view data request
history
NEPA Data Request
> Request form based on
NEPA test case template
> tabular or tree view file
view with file categories
and metadata
Manifest
> data request manifest
at directory and file level
Legend
Stage 1:
September
2014
deployment
3SAQS Steering Committee
October 10, 2014
39