Time Series in Arc Hydro CE 394K GIS in Water Resources University of Texas at Austin Fall 2002 Prepared by: Tim Whiteaker and David R. Maidment Time Series in Arc Hydro 1.Introduction................................................................................................................................................. 1 2. Retrieving and Displaying NWIS Data ..................................................................................................... 2 A. Familiarizing Yourself with the Data .................................................................................................. 2 B. Loading the NWIS tool into ArcMap .................................................................................................. 3 C. Using the NWIS Tool .......................................................................................................................... 4 D. Displaying Time Series in Excel ......................................................................................................... 9 3. Viewing Time Series in a GIS ................................................................................................................. 18 A. Familiarizing Yourself with the Data ................................................................................................ 18 B. Loading the Time Series Viewer into ArcMap .................................................................................. 19 Adding the Arc Hydro Tools ............................................................................................................... 19 Adding the Time Series Tools ............................................................................................................. 19 C. Viewing NEXRAD Data .................................................................................................................... 20 D. Transferring NEXRAD data to Catchments ...................................................................................... 22 E. Performing Rainfall Analyses with the Results ................................................................................. 26 Preparing the Data for the Accumulate Tool ....................................................................................... 26 Running the Accumulate Tool ............................................................................................................. 28 1.1. Introduction Arc Hydro integrates spatial and temporal data in a hydrologic context within a GIS. In this exercise, we look at different ways of utilizing Arc Hydro’s time series format for time series display and analysis. TimeSeries.zip contains the files used for this exercise. The data folder for this part of the exercise can be downloaded from Prometheus or from the web site http://www.ce.utexas.edu/prof/MAIDMENT/giswr2002/ex6/timeseries.zip The files included in TimeSeries.zip are: Guadalupe.mdb: An Arc Hydro geodatabase for the Guadalupe River basin NWIS.zip: A zip file containing NWIS dll and documentation (source files and additional sample data are included for your benefit, but are not required for this exercise). These files are also stored on the LRC Class server at class/maidment/giswr/timeseries. This exercise assumes a basic working knowledge of ArcMap, the ArcGIS Hydro Data Model, and Microsoft Excel. 1 To begin, unzip the contents of TimeSeries.zip to your computer. To start this exercise, open ArcMap, and click on the Add Data button. Browse to Guadalupe.mdb, and add the data in the Hydrography, Drainage, and Network feature datasets. Also add the ts_P_NEXRAD, ts_Q_15min, and TSTypeInfo tables. Explore the data in ArcMap. This database contains data for the Guadalupe River basin in Texas. The database also contains more detailed data (such as Catchments) for the San Marcos River basin, which is located in the northern portion of the Guadalupe River basin. The Basin feature class represents the four USGS hydrologic cataloging units that are within the Guadalupe River basin. You can change the legend for Basin so that each of the four HUC units is labeled and has a different color, for visualization purposes. 1.2. Retrieving and Displaying NWIS Data The USGS publishes real-time and historical water quantity and quality data for monitoring stations in the United States on its National Water Information System (NWIS) web site. Normally, one can retrieve the data by going to the USGS web page and navigating through various web pages. But for this exercise, we will retrieve the data automatically from within an ArcMap application, without ever having to open up a web browser. We will do this with the NWIS tool developed by the Center for Research in Water Resources, CRWR at UT Austin. A. Familiarizing Yourself with the Data For this part of the exercise, turn off all map layers except MonitoringPoint, HydroEdge, and Basin. Open the attribute table for MonitoringPoint. There are 29 MonitoringPoints in this dataset. Each MonitoringPoint represents a USGS stream gage. The 8-digit numbers in HydroCode represent USGS stream gage IDs. The NWIS tool uses the HydroCode to locate the appropriate data online for each gage. 2 B. Loading the NWIS tool into ArcMap Extract NWIS.dll from the Nwis.zip file included with this exercise. You can place the dll in C:\Program Files\ESRI\ArcHydro\bin, or another directory if you wish. Note that the NWIS tool can also be downloaded from http://www.crwr.utexas.edu/gis/archydrobook/ArcHydroTools/Tools.htm. Click ToolsCustomize. Click the Commands tab in the Customize window, and then click Add from file. Navigate to NWIS.dll (it is inside the folder NWIS/dll/as NWIS.dll), and click Open. A window appears showing that clsNWIS was added. Click OK. Note: If you get a message saying ‘No new objects added’, then you may need to have administrator privileges to install the tool. See your network administrator for help. 3 If necessary, click on ‘NWIS’ in the Categories pane of the Customize window. The Get NWIS Data command appears in the Commands pane. To add this button to ArcMap, click on ‘Get NWIS Data’ with the left mouse button. While holding down the left mouse button, drag the Get NWIS Data button next to another button in the gray area where all the commands and tools are in ArcMap. When you see the insertion cursor (looks like a capital I,) that means that tool can be placed in that location. Release the left mouse button to drop the tool in that location. Close the Customize window. The Get NWIS Data button is now ready for use. C. Using the NWIS Tool 4 We will download daily streamflow data for the Guadalupe River at Victoria. To do this, we will select the MonitoringPoint at that location and activate the NWIS tool. The NWIS tool can download data for any number of selected gages, but we’ll just use one in the interest of time for this exercise. In the attribute table for MonitoringPoint, select the record with Name = ‘Guadalupe River at Victoria TX’. This record is now selected in the table as well as the map. Click Get NWIS Data . In the NWIS Data Retrieval window, select MonitoringPoint as the layer, HydroCode as the USGS Site Number field, and HydroID as the Feature ID field. Then click OK. 5 In the NWIS Inputs window, enter 10/1/1999 as the Start Date, and 9/30/2000 as the end date, then click OK. These dates encompass the water year beginning in 1999. In the Select TimeSeries Table dialog box, navigate to our Guadalupe geodatabase and type ts_Q_Daily. Then click Save. This is the table where we will store the USGS daily streamflow data. 6 Input 1 as the TSTypeID. The TSTypeID identifies categories of time series data, e.g. rainfall data might have a TSTypeID of 7, while daily streamflow data might have a TSTypeID of 1. The TSTypeID will link each time series record to a row in the TSTypeInfo table, which provides descriptive information about categories of time series data. It may occur that you are not requested to provide the TSTypeID, and in that event, you’ll have to modify the resulting table to add it later. The tool will now go to the NWIS web site and attempt to retrieve the requested time series data. A window displays the tool’s progress. 7 The tool ignores IDs for which no USGS gage exists. If the available data on the USGS web site isn’t sufficient to cover the entire period of record requested by the user, the tool will only retrieve the data that is available from the web site. When the tool is finished, a message box appears providing some information about the tool’s processing time. Pretty cool! You’ve just access the USGS data server in Reston, VA, at an internet address programmed into this NWIS tool, and retrieved the streamflow data without going through any html web interface at all. In the ArcMap table of contents, note that the TimeSeries (Ts_Q_Daily) and TSTypeInfo tables have been added to the map document. Open these tables and inspect the data. They should look similar to this: 8 If your time series table does not show TsTypeID = 1 (e.g. it has a value of 0 for this field), using the Table Calculator to reset TsTypeID = 1 for the daily streamflow data you’ve just downloaded. Note how the TSTypeInfo table describes the time series data as Daily Streamflow, in units of cfs, at regular intervals, etc. You can use the Statistics tools in ArcGIS table display to define the statistics of a dataset. Right click on the field you want to summarize and select the Statistics option. You’ve now loaded one water year of daily streamflow data into a table in ArcGIS. In the next part of the exercise, we will use Excel’s capabilities to create a graph of time series data for the Guadalupe River at Victoria. To be turned in: What is are the mean, minimum and maximum daily flows (cfs) for the Guadalupe River at Victoria in the 1999-2000 water year (October 1999 – September 2000) ? What is the volume of water (ft3) which passes through the gaging station in this year? D. Displaying Time Series in Excel 9 While ArcGIS is a powerful tool for displaying spatial data, the graphing capabilities of Excel provide a useful tool for displaying certain time series data. Because ArcGIS uses a relational database format to store its data, this data can be imported into Excel. In this portion of the exercise, we will import daily streamflow data for the Guadalupe River at Victoria and plot a graph. Save the ArcMap document, and then close ArcMap. Open Microsoft Excel. Click DataGet External DataNew Database Query… In the Choose Data Source dialog box, click MS Access Database* and then click OK. Navigate to Guadalupe.mdb and click Open. 10 In the Choose Columns window, locate the ts_Q_Daily table, and expand its entry by clicking the + sign next to it. This displays the fields in ts_Q_Daily. To make the plot, we will use the TSDateTime and TSValue fields. Highlight TSDateTime by clicking on it, and then click the > button to move that field to the Columns in your query: text box. Follow the same steps to place TSValue in the window. Click Next. We will not filter the data. In the Filter Data window, just click Next to continue. 11 We will sort the data in chronological order. Choose TSDateTime in the Sort by combo box, and make sure Ascending is selected in the options on the right. Then click Next. In the next window, choose Return Data to Microsoft Excel option, and then click Finish. 12 The next window prompts you for the location where the data will be placed. Cell A1 in the existing worksheet is fine. Click OK. The data are loaded into Excel. Date fields in ArcGIS store date information down to the millisecond, although date values are displayed according to the precision of the value. Thus, even though the data in ts_Q_Daily represent daily records, Excel detects a higher precision, hence the MM/DD/YYYY 0:00 format in the TSDateTime field. 13 To change the TSDateTime field to display daily precision, highlight all of the values in that field, and then click FormatCells. Choose Date as the Category, and 3/14/98 as the type. Then click OK. The data are now displayed with daily precision. 14 To make the plot, select all of the data, and then click the Chart Wizard button. In Step 1 of the Chart Wizard, choose XY (Scatter) as the chart type and select the graph option for just lines (the default is for just points): 15 In Step 2, make sure Columns is selected as the Series in: option. Click Next. 16 In the next two windows, you can customize the look of the plot as you see fit. The finished result may look something like this. 17 You may now save your results and close Excel. Congratulations! You have loaded daily streamflow data from the Internet into ArcGIS, and created a plot of the data in Excel. You are a multi-application master! To be turned in: Make a layout showing the basin, streamflow station, and the excel plot of daily streamflow data (make your own design). 1.3. Viewing Time Series in a GIS Graphs of time series data for a single feature can be plotted quite elegantly in Excel, but when viewing time series for multiple features simultaneously, the GIS may be the better tool for the job. ESRI has developed a time series viewer for accomplishing such a task. The viewer creates a pseudo-animation of time series data by rendering spatial features according to the magnitude of their time series value for a given timestamp while stepping through each timestamp. The time series tool can also transfer time series data from one feature to another. For example, in rainfall data from NEXRAD, Next Generation Radar (a Doppler Radar), all rainfall values are located in NEXRAD cells that can be transferred to catchment polygons, so that each catchment knows how much rain it’s getting. In this portion of the exercise, we will use the ESRI time series viewer to display rainfall data for NEXRAD cells, and then transfer this data to catchments for the San Marcos River basin. A. Familiarizing Yourself with the Data If necessary, open Guadalupe.mxd. For this part of the exercise, turn off all map layers except HydroResponseUnit and Basin. HydroResponseUnit represents the NEXRAD rainfall cells for the San Marcos River basin. Each cell has an hourly rainfall time series associated with it over the period 10/12/2001 11:00 pm to 10/13/2001 2:00 pm. Turn off the Basin map layer. 18 Open the ts_P_NEXRAD table. This table contains the hourly rainfall data for the NEXRAD cells. Each record is linked to a particular cell through the FeatureIDHydroID association. The given rainfall data cover a storm that occurred in October of 2001. Close the table. B. Loading the Time Series Viewer into ArcMap Adding the Arc Hydro Tools You will need both the Arc Hydro toolbar and the Time Series toolbar for this portion of the exercise. If the Arc Hydro toolbar is not present on the ArcMap user interface, click ViewToolbars, and then select Arc Hydro Tools. This displays the Arc Hydro toolbar. You may want to reposition the toolbar for optimum viewing of the data. Arc Hydro Toolbar If Arc Hydro Tools was not one of the options in the toolbars list, you will have to add the toolbar to ArcMap. To do so, click ToolsCustomize. Then click Add from file. Navigate to C:\Program Files\ESRI\ArcHydro\bin, or wherever the ArcHydroTools.dll was installed. Highlight ArcHydroTools.dll and click Open. After a moment, you see a list of new objects added to ArcMap. Click OK to continue. Note: If you get a message saying ‘No new objects added’, then you may need to have administrator privileges to install the tool. See your network administrator for help. Place a check by Arc Hydro Tools to activate the toolbar, and then close the Customize window. Adding the Time Series Tools If the time series toolbar is not present on the ArcMap user interface, click ViewToolbars, and then select TimeSeriesTools. This displays the Time Series toolbar. You may want to reposition the toolbar for optimum viewing of the data. If TimeSeriesTools was not one of the options in the toolbars list, you will have to add the toolbar to ArcMap. To do so, click ToolsCustomize. Then click Add from file. Navigate to C:\Program Files\ESRI\ArcHydro\bin, or wherever the TimeSeriesManager.dll was installed. Highlight TimeSeriesManager.dll and click Open. After a moment, you see a list of new objects added to ArcMap. Click OK to continue. Note: If you get a message saying ‘No new objects added’, then you may need to have administrator privileges to install the tool. See your network administrator for help. Place a check by TimeSeriesTools to activate the toolbar, and then close the Customize window. 19 C. Viewing NEXRAD Data Before we run the time series viewer, we need to set the target locations for storing time series data. To do this, in the Arc Hydro toolbar click ApUtilitiesSet Target Locations. Select HydroConfig as the node to edit, and click OK. 20 Choose Guadalupe.mdb as the database to store vector and time series data, and click OK. To display the time series of rainfall for the NEXRAD cells, click TSProcessingDisplay Time Series. Choose HydroResponseUnit as the layer with time series to display, and ts_P_NEXRAD as the time series table. The tool will then update features in order to efficiently render the features for each timestamp. This may take a few minutes. 21 During this step, the tool adds a field in the HydroResponseUnit table for each timestamp. Each field is populated with the time series value for that timestamp. To display the time series, the tool will create a color ramp for HydroResponseUnit’s legend. For a given timestamp, each feature’s value in the field corresponding to that timestamp is matched up with the appropriate color from the color ramp. This controls the color of the feature in the display. As the time series tool steps through time, the values for each feature change, and thus each feature’s color changes, depending on the magnitude of the value. When the tool is finished updating the features, the Time Series Viewer window opens, and the legend for HydroResponseUnit is automatically updated. You may want to zoom so that you can see the entire San Marcos River basin. Click Start to begin the time series animation. The timestamp is displayed in the top of the map. As time progresses, you can see the storm move from west to east across the basin. If you are working on a slow computer, it may not be able to render the graphics fast enough, resulting in a choppy animation. If this is the case, then pause the animation, zoom in to a smaller area (so that there are less features to render) and start the animation again. D. Transferring NEXRAD data to Catchments 22 It’s neat to watch the storm move across the basin, but we also want to know how much rain we’re getting on a catchment by catchment basis. Thus, we need to transfer the time series data from the NEXRAD cells onto our Catchment polygons. Close the Time Series Viewer Turn off the HydroResponseUnit layer. Turn on the Catchment layer. These are the catchments for the San Marcos River basin. In order to transfer the time series from the NEXRAD cells to the catchments, we first need to intersect the Catchment layer with the HydroResponseUnit layer. This creates an individual polygon for each spatial combination of NEXRAD cells and catchment polygons. The time series tool uses these polygons to determine the average rainfall over a given catchment. For example, at a given timestamp, if 30% of a catchment’s area is covered by a NEXRAD cell with a rainfall value of 1.2 inches, and 70% of the catchment’s area is covered by a cell with a rainfall value of 0.7 inches, then the average rainfall over that catchment for that timestamp is calculated as 0.3*1.2 inches + 0.7*0.7 inches = 0.85 inches. The time series tool processes the NEXRAD cells and catchments in this manner until each catchment has a composite time series value for each time stamp. Once each catchment has its composite value, the intersected layer is no longer needed. To intersect the catchments with the NEXRAD cells, click ToolsGeoProcessing Wizard. In the GeoProcessing Wizard, click Intersect two layers. Click Next. Click the Input layer dropdown arrow and click Catchment. Click the Polygon overlay layer dropdown arrow and click HydroResponseUnit. Save the output in the Hydrography feature dataset of the Guadalupe geodatabase. Call the output intCatchNEXRAD. 23 Click Finish. It may take a few minutes to intersect the features. Now is a good time to stand up, stretch, exercise your eyes, and relax a bit. The intersected layer appears in the map. Next, click TSProcessingIDTransfer. Enter HydroResponseUnit as the From Layer, Catchment as the To Layer, and intCatchNEXRAD as the Intersect Layer. Click OK. Enter HydroID as the Key Field for each layer. Click OK. 24 It may take a few minutes for the tool to process the features. In this step, the tool is figuring out what percentage of each NEXRAD and Catchment feature is represented by each intersected feature. It will use these percentages to determine the composite time series value for each Catchment. When the tool is finished with the ID transfer, click TSProcessingValueTransfer. Enter Catchment as the To Layer, intCatchNEXRAD as the Intersect Layer, ts_P_NEXRAD as the Time Series table, and ts_P_Catchment as the Target TSTable. Click OK. Choose 7 as the TSType. This is the TSType we use for rainfall data in this exercise. Click OK. In this step, the tool is creating a new time series table for the catchments. The values in this table are the weighted average value of the rainfall for each catchment at each timestamp. It may take a few minutes to process the features. Now is a good time to check your email, or research the things you can do to live more sustainably. When the tool is finished processing, click TSProcessingDisplay Time Series. Choose Catchment as the layer with time series to display, and ts_P_Catchment as the time series table. The tool will then update features in order to efficiently render the features for each timestamp. This may take a few minutes. When the tool is finished processing, the legend for Catchment will be updated, and the Time Series Viewer window will appear. If necessary, turn off the intCatchNEXRAD layer. Click Start to begin the animation. Note how the rainfall is being rendered onto the Catchments now. There are a lot more catchments than NEXRAD cells, so it takes more computing power to render them. If necessary, zoom in to a smaller area to produce a smoother animation. 25 With the Time Series Viewer, you could also transfer the time series to MODFLOW cells for groundwater modeling, or to any other polygonal feature class. You can customize the legend for rendering by playing with parameters in the Time Series Viewer. When you are finished viewing the data, close the Time Series Viewer. TIP: To make the Timestamp that appears at the top of the map disappear, uncheck the Show Text check box on the Time Series Viewer window. Then close the viewer. E. Performing Rainfall Analyses with the Results With the standard Arc Hydro tools, we can perform some useful analyses with our time series data. In this portion of the exercise, we will determine the total rainfall over the San Marcos River basin using the Accumulate tool. Preparing the Data for the Accumulate Tool Now we will prepare the data for the Accumulate tool. Open the attribute table for Catchment. These catchments were processed from raster data using the Arc Hydro tools. Thus, the NextDownID attribute is populated. This ID points to the HydroID of the next downstream Catchment. We can use this attribute to accumulate rainfall totals from upstream catchments to downstream catchments. But first we must add a field to store the total rainfall. Note that a join exists between Catchment and another table called TSCatchment7. This join was used by the Time Series Viewer. We will use this join to calculate the amount of rainfall that fell on each catchment. In the attribute table for Catchment, click OptionsAdd Field. Create a field called incRainfall of type Double. For each catchment, this field will store the rainfall that fell directly on that catchment. The field will appear within the Attribute table for the Catchment feature class as Catchment.incRainfall. It doesn’t appear at the end of the table since the join table TSCatchment7 has its fields stored there. TIP: If the Add Field option is not active, then you might be in an edit session. You cannot add fields if you are in an edit session. If necessary, Click EditorStop Editing, and then try adding the field. If you 26 still cannot add a field, save your Map file, close ArcMap, go to the folder where your geodatabase is stored and delete a small 1KB file that is called Microsoft Access Record-Locking Information, open ArcMap back up again, and proceed. In the attribute table for Catchment, right click on the incRainfall field. In the context menu, click Calculate. When given the alert about performing this operation outside of an edit session, click Yes. In the calculate text box, input the sum of all of the render fields that the Time Series Viewer added. These fields store the time series value for each timestamp, so the sum of these fields represents the total amount of rainfall for the entire storm. The resulting expression looks like this: [TSCatchment7.TSFLD1] + [TSCatchment7.TSFLD10] + [TSCatchment7.TSFLD11] + [TSCatchment7.TSFLD12] + [TSCatchment7.TSFLD13] + [TSCatchment7.TSFLD14] + [TSCatchment7.TSFLD15] + [TSCatchment7.TSFLD16] + [TSCatchment7.TSFLD2] + [TSCatchment7.TSFLD3] + [TSCatchment7.TSFLD4] + [TSCatchment7.TSFLD5] + [TSCatchment7.TSFLD6] + [TSCatchment7.TSFLD7] + [TSCatchment7.TSFLD8] + [TSCatchment7.TSFLD9] 27 Click OK. If you get a message that says VB Error, use View/Toobars/Editor to open an Edit session, then select Modify Feature for the Catchment feature class, then proceed as above (this is to get around the presence of that Microsoft Locking file again). The above operation gives the total rainfall depth in inches for the storm. If in an edit session, use Editor/Stop Edit to close the edit session and save your new incremental rainfall data. To be turned in: a map showing the depth of rainfall in inches that fell on each catchment throughout the storm. Volume of Rainfall In the attribute table for Catchment, click OptionsAdd Field. Create a field called volRainfall of type Double. For a given catchment, this field will store the total rainfall that fell on that catchment, plus the rainfall that fell on all upstream catchments. However, for this exercise we’re concerned with the total volume of rainfall, so we need to multiply the rainfall depth by the area of each catchment to obtain the volume. Right click on volRainfall again and click Calculate. This time, we will multiply the rainfall depth by the catchment area and a conversion factor. In the calculate text box, input [Catchment.incRainfall] * [Catchment.AreaSqKm] *39370 39370 is a conversion factor, so that the output will be in units of cubic meters. Click OK. Each catchment now knows the total volume of rainfall that fell directly on it for the given storm. Running the Accumulate Tool 28 We’ll find the total rainfall for the San Marcos River basin by running the Accumulate tool on the outlet catchment for the basin. This catchment has a HydroID of 12617. To select this catchment, either use the Selection tool or perform an attribute query for HydroID = 12617. In the attribute table for Catchment, click OptionsAdd Field. Create a field called accRainfall of type Double. On the Arc Hydro toolbar, click Attribute ToolsAccumulate. Click Feature Layer with NextDownID as the Trace Type. Choose Catchment as the Feature Layer with NextDownID. Choose NextDownID as the Next Down ID Field. Make sure Use selected features only is checked. Choose Catchment as the Source Feature Layer. Choose volRainfall as the Source Field. Click Sum as the Accumulation Type. Don’t use a weighted average Choose Catchment as the Target Feature Layer. Choose accRainfall as the Target Field. Click OK. 29 The Accumulate tool uses the NextDownID—HydroID association to determine which catchments are upstream of the outlet catchment. It then sums the rainfall totals (from the incRainfall field) from those catchments and adds it to the rainfall of the outlet catchment. When the tool is finished, use the Identify tool to click on the outlet catchment. Note that a total of 270,788,991 m3 of rain fell on or upstream of this catchment. To be turned in: What is the average depth of rainfall during this storm over the whole San Marcos basin covered by these catchments? Express the result in mm and in inches of rainfall (1 inch = 25.4mm) 30 Congratulations! You have produced some cool time series animations in a GIS, and used NEXRAD data to determine the amount of rainfall each San Marcos catchment received during a storm. You have also determined the total depth of rainfall for the San Marcos River basin for a given storm. Pretty cool stuff! Summary of items to be turned in: (1) What is are the mean, minimum and maximum daily flows (cfs) for the Guadalupe River at Victoria in the 1999-2000 water year (October 1999 – September 2000) ? What is the volume of water (ft3) which passes through the gaging station in this year? (2) Make a layout showing the basin, streamflow station, and the excel plot of daily streamflow data (make your own design). (3) A map showing the depth of rainfall in inches that fell on each catchment throughout the storm. (4) What is the average depth of rainfall during this storm over the whole San Marcos basin covered by these catchments? Express the result in mm and in inches of rainfall (1 inch = 25.4mm 31