New Tools for a Better
Air Quality System Data Flow
EN 2012
May 31, 2012
Topics to Cover
• Air Quality System (AQS) flow automation
enhancements
• Exchange Network Services Center
• AQS Plugins for OpenNode2 and EN-Node
• Q&A / Discussion
Jumping Through Hoops
Can I interest you in
an AQS data flow?
• AQS submissions via a Node or Node Client used
to require extra manual steps
• More difficult to access AQS submission logs and
reports
• Most users preferred to use EPA’s non-Exchange
Network interface called CDX-Web
Fewer Flaming Hoops…More Puppies
Perhaps this is more
to your liking?
• Today, files submitted through the Exchange Network are
automatically processed by AQS
• Exchange Network Nodes can download the submission log
and reports
• New tools for submitting AQS data via the Network
• Support for AQS submissions through CDX Web is being
discontinued
How Does it Work?
• AQS data flow (v2.2a) now uses the Exchange
Network Header
• The Header file contains information about the
submission data file
– Security Token
– Exchange Network and AQS User IDs
– Payload type: 1) XML or 2) FLAT
– AQS Load Options
• Final Processing Step
• Stop on Error
AQS Load Options
The submitter defines the desired level of automation:
1. Stage: Submission file is uploaded to the AQS server
and verified to be a valid AQS file type, but with no
subsequent processing.
2. Load: The file is loaded into the AQS database and its
contents validated. Sample measurements are left at
the status “Pre-production”; all other data types (Site
information, Monitor information, and QA
information) are set to the status “Production”.
3. Post: All valid sample measurement data for the file is
set to the status “Production”.
–
Post is the default setting
New Submission Option #1
• Exchange Network Services Center
https://enservices.epa.gov/login.aspx
– Automatically creates EN Header
– Supports manual submissions to AQS
– Replacing CDX-Web for AQS submissions
New Submission Option #2
Node Plugins for OpenNode2 and EN-Node
•Customized for easy integration with 2 widely used air
monitoring systems
–
–
–
–
OpenNode2 - .NET – AirVision and DR-DAS
OpenNode2 - Java – DR-DAS
EN-Node - .NET – AirVision
EN-Node – Java - AirVision
•Automatically create EN Header
•Support submissions to AQS and data publishing
•In testing. Production-ready versions likely available in
June for free download from EN website.
Show and Tell
• Exchange Network Services Center
• EN-Node Plugin
– AirVision
• OpenNode2 Plugin
– AirVision & DR-DAS
EN-Node AirVision Plugin
• AirVision Plugin Features  3 Operations
1
2
3
Submit AQS data to CDX
Publish AQS data (Solicit)
Publish AQS data (Query)
ENDS
Solicit
2
Query
3
EN Browser
EPA
AQS
CDX
EN-Node
AQS
Monitor
AirVision
Plugin
1
Submit
AirVision
Installation File & Contents
• URL: http://code.google.com/p/en-node2/downloads/list
• File: AirVisionPlugIn.zip
File Name
Purpose
Installation
1. Six (6) DLL files
Plugin software
Copy to EN-Node
file directory
2. InsertAQSHeaderTemplate.sql
DB scripts to install
AirVision Plugin in
EN-Node DB
Execute scripts
3. Task_SubmitAQSXMLData_DataWizardConfig.xml Submit AQS data to
CDX
Copy to EN-Node
file directory
4. Query_GetAQSXMLData_DataWizardConfig.xml
Publish AQS data
(Query)
Copy to EN-Node
file directory
5. Solicit_GetAQSXMLData_DataWizardConfig.xml
Publish AQS data
(Solicit)
Copy to EN-Node
file directory
Configure AirVision Plugin
• What is Configuration?
– Business logics used by EN-Node to complete the data flow
– Config. data saved in an external BEPL file (Business Process Execution
Language)
– Each Operation has its own BEPL file
– Configuration tools
• EN-Node Admin Console
• BEPL XML file provided with the Plugin installation file
• How to Configure:
Step 1: Create a Domain (AirVision)
Step 2: Register 3 operations under AirVision
1. Submit AQS data to EPA
2. Publish AQS (Query)
3. Publish AQS (Solicit)
Step 3: Test Plugin
Step 4: Publish AQS data at ENDS or EN Browser
Data Needed to Configure AirVision Plugin
Plugin Configuration Options
Parameter
Required?
Value
Example
EPAURL
Yes
URL of the EPA Endpoint in which
to submit data
https://testcdxnode64.e
pacdxnode.net/cdxenws20/services/Netwo
rkNode2
NAASUserID
NAASUserPassword
Yes
Yes
User Name
Password
AirVisionURL
Yes
URL of the AirVision Node
FileName
No
File Name
Enfotech
enfopassword
http://localserver/AirVis
ion.WebService
Airvision_Submission
AQSXMLSchemaVersion
No
Version Number
2.2
CompressPayload
DurationCode
No
No
True/False
Number in hours
True
1
StartTime
Yes
Date in the format YYYY-MM-DD
2011-10-01
EndTime
Yes
Date in the format YYYY-MM-DD
2011-12-31
SendOnlyQAData
SendRATransactions
SendRBTransactions
SendRDTransactions
SendRPTransactions
AuthorName
OrganizationName
DocumentTitle
AQSUserID
AQSScreeningGroup
AQSSchemaVersion
No
No
No
No
No
No
No
No
No
No
No
False
True
True
True
True
John Smith
Org1 Inc.
AQS Sample
Enfotech
Enfotech
2.0
AQSPayloadTypeProperty
No
AQSLoadOptions
PayloadOperation
AgencyCode
SiteCode
ParameterCode
No
No
No
No
No
True/False
True/False
True/False
True/False
True/False
Author Name
Organization Name
Document Title
AQS User Identifier
AQS Screening Group
Schema version of document
Format in which data is
presented
Load/Operation
Load/Operation
Agency Identification Code
Site Identification Code
Parameter Code
XML
Load
Load
001
210
46324
Minimum Data Needed
•
•
•
•
•
AirVision End Point
EPA AQS End Point
Organization Name
NAAS ID & PWD
Plugin file location
Before Configuration…
Important!
•
•
Open 3 DataWizardCinfig.xml files with text editor
Enter Plugin file location
Step 1: Create Domain
• Create AirVision domain
– Choose web services (Submit, Download, Query, Solicit, Notify) allowed for
the domain.
• Establish “Administrator” accounts to setup/modify this domain.
Step 2: Register Operation(s)
• Register Operation(s)
1. Submit AQS data to EPA: Task_SubmitAQSXMLData_DataWizardConfig.xml
2. Publish AQS (Query): Query_GetAQSXMLData_DataWizardConfig.xml
3. Publish AQS (Solicit): Solicit_GetAQSXMLData_DataWizardConfig.xml
Define
•EPA Endpoint URL
•Validation rule (if any)
•Data viewing stylesheet (if any)
Define
•
Task schedule
Step 2: Register Operation(s)
• Register Operation(s)
1. Submit AQS data to EPA: Task_SubmitAQSXMLData_DataWizardConfig.xml
2. Publish AQS (Query): Query_GetAQSXMLData_DataWizardConfig.xml
3. Publish AQS (Solicit): Solicit_GetAQSXMLData_DataWizardConfig.xml
Step 3: Test Operation(s)
1. Test Submit AQS to EPA
– Test: Enter Start Date and End Date to a
date earlier than the current date will
execute task after the user clicks save.
– Track submission status




CDX will return “Completed” or “Failed”
View
Download
Data Viewer
Step 3: Test Operation(s)
2. Test AQS Data Query
– Authenticate
– Input Start Date and End Date for AQS
monitoring date
– Query will return data in XML
Authentication
Query Request
Step 3: Test Operation(s)
3. Test AQS Solicit
– Authenticate
– Solicit (Input AQS monitoring date)
– Get Status
– Download AQS data (in XML)
Authentication
Solicit Request
Get Status
Call Download
Step 4: Publish AQS Data at ENDS/ENB
• Manual data input to ENDS:
– Test: https://ends2.epacdxnode.net
– Production: https://ends2.epa.gov
• XML Data Upload to ENDS:
– Auto Pull from ENDS: GetServices
– Manual XML Upload to ENDS
•
•
•
•
Test: https://ends2.epacdxnode.net/Node2WS.svc
Production: https://ends2.epa.gov/Node2WS.svc
Dataflow: ENDS_v20
Operation: Service Refresh
ENDS: Service/DEDL Registration
Option A: Manual
Node Admin Registers
A1 Services (ENDS)
Data Provider
A2
Node Admin Registers
DEDL
Node Admin
Exchange
Network
Node
EPA Service
Registry
(ENDS)
Option B: Automated
(Automate your Node to transfer
registration info to EPA’s Registry)
B1
ENDS XML
B2
DEDL XML
(Published via
GetServices)
(Node submits
DEDL to EPA)
Stores registry of all
services, input
parameters, pick lists, etc.
EN Browser
Useful Tips for EN-Node AirVision Plugin
• Implement Only the Operation(s) you need
• Minimum data you need to configure Plugin
–
–
–
–
–
AirVision End Point
EPA AQS End Point
Organization Name
NAAS ID & PWD
Plugin file location
• Useful resources
– Plugin: URL: http://code.google.com/p/en-node2/downloads/list
– Data Publication Guide:
http://www.enbrowser.net/enbrowser/ClientBin/HowToPublishData.pdf
– ENDS & FCD: http://www.exchangenetwork.net/2011/07/15/exchangenetwork-discovery-service-ends/
• Support
– enfoTech, Lijie Xu: lijie_xu@enfotech.com
Show and Tell:
OpenNode2 AQS v2.2 Plugin
OpenNode2 AQS v2.2a Plugins
• http://code.google.com/p/opennode2
• .NET OpenNode2
– AirVision
– DrDAS
• Java OpenNode2
– DrDAS
OpenNode2 AQS Submit Workflow
AirVision
Or
DrDAS
Web
Service
1. Request Data
OpenNode2
3. Submit Data
2. Return Data
1. On a set schedule (or manually triggered),
request data from AirVision or DrDAS
•
Parameters set to filter data request
2. AQS XML data is returned to OpenNode2
3. Relay the data to EPA CDX/AQS system
•
•
Optionally, header is added
Optionally, notifications can be sent
EPA
CDX
Node
Get Status/Download Workflow
1. Get Status
Status Response
OpenNode2
3. Notifications
2. Download Request
EPA
CDX
Files
1. On a set schedule (or manually triggered),
check status of outstanding submissions
2. If CDX returns “completed” or “failed”,
download processing reports from CDX
•
Do nothing if status is “pending”
3. Optionally, send notification emails to agency
staff
1. Mail sent for all submissions where status
changes from “pending” to “completed” or
Publishing AQS Data from OpenNode2
• “Publishing” makes data available for any
authorized partner to come and query from
your node
– Exposed as Query and/or Solicit services
• No additional setup needed to make these
services available from OpenNode2
– Setting up the services in Node Admin tool makes
them available with a simple checkbox
Setting up the AQS Plugin/Flow
• Very similar to setting up any other
OpenNode2 data flow
• Using the Node Admin interface:
1. Create the AQS flow
2. Upload the plugin (contains all workflow logic)
3. Configure the Services (exposes plugin logic)
a)
b)
Service 1: Get data from AirVision/DrDAS for submission to EPA
Service 2: Check status of submission and download results
4. Create the Schedules (automates workflow)
… and that’s it for the OpenNode2 portion
OpenNode2 AQS Service Setup
View in Node Admin of AQS flow with services configured:
Flow 1
Service 1
Service 2
Flow 2
Service 1
Service 2
AQS “GetXmlData” Service Detail
Note: AirVision settings shown
DrDAS settings very similar
AirVision Schedule Detail
DrDAS Schedule Detail
Parameter
Value
FileGenerationPurposeCode
“AQS”
BeginDate
Desired monitoring period start date
BeginTime
“00:00”
EndDate
Desired monitoring period end date
EndTime
“23:59”
DataValidityCode
“V”
DataApprovalIndicator
“Y”
IncludeMonitorDetails
“N”
IncludeEventData
“FALSE”
SchemaVersion
“2.2”
Questions?
Kurt Rakouskas
301.531.5186
kurt@exchangenetwork.net