Guidelines for State and Local Submissions to ICIS-Air STEP 1 - IDENTIFY STATE OR LOCAL DATA FOR SUBMISSION TO ICIS-AIR ICIS-Air is very different from AFS. Always map the data from your state source database into the ICIS-Air XML tags instead of trying to map the AFS batch cards into ICIS-Air XML tags! Suggested Materials to Use (posted at http://www.exchangenetwork.net/edt-ipt/ ): ICIS-Air Plug-in Staging Tables List of ICIS-Air MDRs ICIS-Air Example XML Instance Document ICIS-Air Electronic Data Transfer Technical Specifications ICIS-Air Web Design Documents Migration Mapping of AFS Fields into ICIS-Air Fields Migration Mapping of AFS Reference Table Codes into ICIS-Air Reference Table Codes ICIS-Air Reference Table Code Lists ICIS Data Exchange Template ICIS-Air Electronic Data Transfer User Guide 1. If using an ICIS-Air Plug-in, download staging tables from the Exchange Network at www.exchangenetwork.net/exchange-network-products/ or request them from your Node software provider: OpenNode2 states must use staging tables developed for OpenNode2 ICIS-Air plug-in only: o Draft version of these tables ready in May 2014 o Final version of these tables ready in July 2014 ENNode states must use staging tables developed for ENNode ICIS-Air plug-in only: o Date not yet known when tables will be ready Virtual Node locals and states must use staging tables developed by CDX o In progress - contact Alison Kittle or Jeff Clark to get these tables OR 1. If using customized code instead of an ICIS-Air plug-in, use the Alabama DEM ICIS-Air mapping spreadsheets, or design your own using the information in the mapping tables found in Section 8 of the ICIS-Air Electronic Data Transfer User Guide as illustrated below. 1 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air 2. Map data from state or local system into staging tables: Identify mandatory and repeatable XML tags to be submitted by reviewing the ICIS-Air Example XML Instance Document: o Examples of Replace and Mass Delete transactions for each data family with all possible mandatory and optional tags that can be reported o Tags that must be present in the XML are shown in green-colored font with bold typeface, black-colored font with bold typeface, and, if its parent tag is present then all tags in blue-colored font o Tags that do not need to be present are shown in black or green with non-bold typeface o Tags that are repeatable are shown in green-colored font – these tags must be repeated consecutively Identify mandatory air compliance and enforcement data tags to be provided using the List of ICIS-Air MDRs plus any extra data elements specified in submission agreements between your agency and your EPA region: o List of ICIS-Air MDRs to be ready May 1, 2014 Use the ICIS Data Exchange Template spreadsheet to understand the description, format, and name of the ICIS-Air reference table for each tag to be submitted 2 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air 3. Identify special rules for converting your mapped data fields into ICIS-Air format: If your system is similar to AFS, consider using the same rules as EPA is using to convert the AFS data into ICIS-Air as documented in the AFS Migration Rules spreadsheet that is updated and re-distributed after each AFS migration test iteration: o One tab for each ICIS-Air data family lists the AFS fields being migrated and how they are being migrated o ICIS-Air table and column names are on the left under the gray header area, AFS table and field names are in the middle under the blue header area, and migration information including exception and information messages are on the right under the olive header area Refer to the ICIS-Air EDT Technical Specifications first, then the ICIS-Air Web Design Documents and to understand how incoming data will be processed and what will cause data to be rejected. Pay special attention to the business rule error messages near the end of each ICIS-Air EDT Technical Specification document to understand what will be rejected and why 3 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air 4. Identify special rules for converting your reference table codes into ICIS-Air reference table codes: If your system is similar to AFS, consider using the same rules as EPA is using to convert the AFS data into ICIS-Air as documented in the AFS Code Mapping into ICIS-Air Codes spreadsheet that is updated and re-distributed after each AFS migration test iteration: o One tab for each reference table code lists rows, each row containing one existing AFS code with the code it is being migrated to in ICIS-Air o AFS codes are listed on the left under the light gray header area, their corresponding ICIS-Air codes are listed on the right under the green header area Use the spreadsheets in the zipped ICIS-Air Reference Code Tables file to find the most applicable code in ICIS-Air that matches the code in your state system: o Updated ICIS-Air reference table code lists are distributed with each round of pilot testing, user validation testing, and user acceptance testing o Do not use inactive codes o Some reference tables have a flag that makes a code applicable to delegated agencies versus EPA regions, or a specific data family – the design document for each data family will provide this information, or the error messages in the technical specifications for each data family will identify them for you o If you cannot find an ICIS-Air code that matches your code contact Jason Swift to have it added to ICIS-Air 4 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air 5. Refine your data mapping by identifying the ICIS-Air rules that will cause data to be rejected or stored improperly: Refer to the ICIS-Air EDT Technical Specifications first, then the ICIS-Air Web Design Documents and to understand how incoming data will be processed and what will cause data to be rejected. Pay special attention to the business rule error messages near the end of each ICIS-Air EDT Technical Specification document to understand what will be rejected and why 5 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air STEP 2 - CREATE EXTRACTS OF STATE OR LOCAL DATA FOR SUBMISSION TO ICIS-AIR If you are using an OpenNode2 or ENNode ICIS-Air Node Plug-in, ask your contractor representative if they have generic extraction SQL templates that you can use! Suggested Materials to Use: ICIS-Air Plug-in Staging Tables Mappings of state data into ICIS-Air fields performed above 1. Create staging tables in database of your choice using your mappings performed above, if not using an ICIS-Air plug-in 2. Create extracts to pull data out of state or local system in ICIS-Air format and place the data into staging tables These will typically be SQL statements or data export/imports to be run against the state or local data Build extract code to pull data out of state system, transform the field into ICIS-Air format and substitute the state system reference table codes into the ICIS-Air codes using the work performed in steps 2 through 4 above, and store the data into the staging tables OR Extract raw data out of state system into a local database, run PL/SQL, Java, etc. code to transform the field into ICIS-Air format and substitute the state system reference table codes into the ICIS-Air codes using the work performed in steps 2, 3 and 4 above, and store the data into the staging tables 3. If not using OpenNode2 ICIS-Air plug-in, make sure data extracts include a way to extract data that needs to be added or changed in ICIS-Air and your extract can provide an “R” as a transaction type for these records in your staging tables, and that your data extracts include a way to extract data the needs to be deleted from ICIS-Air and your extract can provide an “X” as a transaction type for these records in your staging tables OR 3. If using an OpenNode2 plug-in to submit a data family to ICIS-Air for the first time, use their ICISAir data extractions to populate the ICIS-Air staging tables 6 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air STEP 3 - PREPARE SOFTWARE TO PROVIDE STATE OR LOCAL DATA TO ICIS-AIR IN XML FORMAT ICIS-Air can process one XML containing only one data family, or one XML containing multiple data families. See Sections 11 and 12 of the Example XML Instance Document for details! Suggested Materials to Use: ICIS-Air Plug-in ICIS-Air Example XML Instance Document ICIS-Air Electronic Data Transfer User Guide ICIS Data Exchange Template Zip Compression Software such as Winzip 1. Download free ICIS-Air plug-in from Exchange Network for your Node OR 1. Refer to the ICIS-Air Example XML Instance Document XML templates, ICIS Data Element Dictionary tag descriptions, and ICIS-Air Electronic Data Transfer User Guide instructions to create routines that take the data from the staging table and convert it into the tags and XML submissions needed for ICIS-Air: Tags must in the same order as shown in the templates (<FirstName> comes before <MiddleInitial>, etc.) , and tag names have the same spelling and capitalization as shown in the template (<LastName> is acceptable but <lastname>, <Lastname> and <LASTNAME> are not) The ID tag must contain an ICIS-Air WAM ID that has the rights to add, change and delete records for the data family being submitted Only one name/value pair of “Source/Fullbatch” can exist for a parent Property tag Multiple name/value pairs of “e-mail/<emailaddress>” can exist for a parent Property tag 2. Obtain IDs for being able to submit data to ICIS-Air: 7 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air WAM ID = used to log into ICIS-Air and must be provided in the <Id> tag of XMLs to be submitted, and must have roles for adding/changing/deleting data for your agency in ICISAir – Regional Air Compliance Managers assign WAM IDs NAAS ID = used to submit data to EPA using a Node or the EN Services Center web form – Node Help Desk assigns NAAS IDs for anyone using the EN Services Center web form to manually upload XML submissions to ICIS-Air, or Node Help Desk maps a state Node’s existing NAAS ID into ICIS-Air 3. If not manually uploading submissions, install and configure ICIS-Air Plug-in or your agency’s customized software to use appropriate WAM IDs and email addresses in XML submissions it generates, and point to the correct endpoint URL and dataflow name: Request endpoint URL and Dataflow Name from Alison Kittle or Jeff Clark 8 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air STEP 4 – TEST SUBMISSIONS OF STATE OR LOCAL DATA TO ICIS-AIR IN XML FORMAT A “submission” is considered to be one zipped file that can contain one or more XML files. CDX and ICIS-Air will attempt to process all XML files within a submission in random order! ICIS-Air will process records within an XML in chronological order only if the TransactionTimestamp tag is populated with the date the record has been added or changed! Suggested Materials to Use: ICIS-Air Plug-in ICIS-Air Electronic Data Transfer User Guide Zip Compression Software such as Winzip ICIS-Air Electronic Data Transfer User Guide ICIS-Air Electronic Data Transfer Technical Specifications ICIS-Air Web Design Documents 1. Extract data from your state system into the staging tables 2. Kick off the ICIS-Air plug-in OR 2. Zip up the XML to be submitted and run your agency’s customized submission code OR 2. Zip up the XML to be submitted and log into the Exchange Network Services Center and manually upload the zipped file: Refer to Section 6.1 of the ICIS-Air Electronic Data Transfer User Guide for instructions on how to manually upload files 3. Review ICIS-Air results using the instructions in Section 7 of the ICIS-Air Electronic Data Transfer User Guide: If using an ICIS-Air plug-in, use plug-in or your email to see results If using agency customized code for submissions, coordinate with IT staff on viewing your results If manually uploading submissions, follow the instruction in Section 7 for downloading the results 9 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air Log into ICIS-Air, go into Business Objects, and run Batch Audit reports for any submissions for your agency within a timeframe you specify 4. Log into the ICIS-Air web application and use the Search screens to find records from your submissions and browse them to make sure all fields were saved properly. 5. Look at the error messages for records in the Rejected Transactions results report, fix then in your extracts or within your state or local system: Refer to Section 9 of the ICIS-Air Electronic Data Transfer User Guide or the ICIS-Air EDT Technical Specification document for each data family to find an error message and discover why it occurred Refer to the ICIS-Air Web Design documents and ICIS-Air EDT Technical Specifications for questions you may have on why data is stored in ICIS-Air a certain way. 10 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air STEP 5 – REPORTING SUBMISSION PROBLEMS All incoming submissions to ICIS-Air are assigned a “Transaction ID” by CDX. This Transaction ID is needed to troubleshoot any submission problems you may encounter! Submission results from ICIS-Air are returned to the same tool that was used for a submission. You cannot log into the EN Services Center to see results from your Node’s submissions! 1. Use the table below to contact the appropriate people for assistance: Node Help Desk ICIS-Air Team State or Virtual Node access problems ICIS-Air Online Web System, Batch Audit report, and Business Objects Ad Hoc problems or questions Exchange Network Services Center web login problems XML file validation errors (submission status of “Failed” and submission-meta.xml does not contain the word “Congratulations” in it.) State Node, Virtual Node, or Exchange Network Services Center password resets ICIS Batch Audit accepted, rejected, and transaction summary report problems or questions ICIS-Air Regional Air Coordinators ICIS-Air WAM ID requests, and WAM ID password resets XML file transfer problems 2. For ICIS-Air WAM ID assistance, visit http://www.epa.gov/compliance/contact/data-icis.html to find your Regional Air Coordinator 3. For ICIS-Air Team assistance, send an email to kittle.alison@epa.gov and clark.jeffreyf@epa.gov and provide the following information: Attach your XML submission to the email Attach submissions-metadata.xml to the email if it does not say “Congratulations” in it Date of Submission Transaction ID of Submission 11 5/1/2014 Guidelines for State and Local Submissions to ICIS-Air Status of Submission Steps followed leading up to the submission What you expected to happen What actually happened 4. For Node Help Desk assistance, send an email to nodehelpdesk@epacdx.net, kittle.alison@epa.gov and clark.jeffreyf@epa.gov and provide the following information: Date of Submission Transaction ID of Submission Status of Submission Steps followed leading up to the submission What you expected to happen What actually happened 12 5/1/2014