Using Python to implement a Hydrologic Information System for Texas James Seppi (CRWR, University of Texas at Austin) Dharhas Pothina (Texas Water Development Board) Ernest To (Espey Consultants, Inc) Tim Whiteaker (CRWR, University of Texas at Austin) Andrew Wilson (Texas Water Development Board) Texas Water Development Board Provides leadership and scientific expertise for development and conservation of water Accumulate and publish large-scale State water databases in a form accessible as WaterOneFlow web services. Create a Texas-centric HIS Water Data for Texas: Partnerships with Federal and Texas agencies to obtain data Inherits National HIS technology Adds Texas-specific services to provide ease of use and a level of quality control Why a Texas-specific HIS? National HIS technology available but uptake low among Texas agencies lack of motivation lack of time/resources lack of experience with the technology National HIS doesn’t quite do everything TWDB needs. Parameter mapping Unit Conversions Web Viewer original paradigm Put your data in an ODM database i.e. learn swahili this didn’t get much traction… Data is usually already in the format the organization needs. Changing formats/delivery mechanisms is a pain. ODM DB Schema doesn’t always fit the needs of data collecting agencies. Resources. Resources. Resources. Overall response: yawn new paradigm… make it easy Form partnerships to obtain data Provide technical know-how and support Provide multiple paths to serve data Provide a useful product with branding that is available to the general public can use. Scheduled updates to datasets i.e. we have trained swahili translators available please speak to us Four Components of 2. Translation services to convert nonWaterML data sources into WaterML 3. WDFT Central Registry service Limited Parameter Set List of Texas HIS data sources Unit conversions 1. Existing WaterML compliant web services within Texas WaterML compliant web services 4. Access Tools for data discovery and download: HydroExcel, HydroDesktop, HydroGET… GEMSS (TNRIS hosted map based web interface) Why python? Built-in cross-platform compatibility Availability of a massive variety of scientific libraries/modules Amazing community support. Better than commercial vendors $FREE!$ Python @TWDB pINT - Python INstrument Toolkit SWIS - Surface Water Information System WOFpy - Python WaterOneFlow Wrapper pyHIS - Python HIS Client pyHAT - Python Hydrographic Analysis Toolkit pyselfe/pug – hydrodynamic model analysis & visualization WOFpy concept WOF py WOFpy – a python wrapper for WaterOneFlow Services Old paradigm (i.e. ODM db as intermediate) Data in ODM Native database ODDataloader SSIS scripts SQL views Manual loading, etc. WaterML Generic ODwebservices •Significant learning curve •Data persists in ODM under Microsoft SQL server i.e., not in native format and environment •“When is the next update?” •“What if data is collected in real-time?” Old paradigm (i.e. ODM db as intermediate) Metadata in ODM WaterML for GetSites, GetVariables, Native metadata Loading scripts Realtime data on web etc. Generic ODwebservices WaterML for GetValues Customized webscraper web service Old paradigm (ODM db asframework) intermediate) New paradigm (i.e. single Metadata in ODM Single framework WaterML for GetSites, • Dynamically map between native database GetVariables, schema and WaterML Native etc. • Allow data to persist in native database metadata • Accomodate different database environments WaterML (MSSQL, SQLite, etc) • Handle both static and real-time data • Dispatch and manage WaterOneFlow services in both SOAP WOF py Generic Loading scriptsand REST ODwebservices • Open-source and fully customizable WOFpy – a python wrapper for WaterOneFlow Services Realtime data on web WaterML for GetValues Customized webscraper web service WOFpy architecture (backend – data access) Data persists in source’s native db environment Native metadata database WOF py (WOF module) WOFpy – a python wrapper for WaterOneFlow Services • WOF module assembles WaterML elements to WaterML responses. • Implements a reduced ODM data model (‘Model’) for mapping to WaterML elements. • Model is also mapped to the native db of the source by source-specific DAOs. Source-specific DAOs Realtime data on web (Data Access Objects) WaterML classes • Model DAOscontains define object-relational mapping (ORM) common to both from native schema to Model. WaterML ODM.for translating user • Contains and methods arguments into queries to native db. • The python module, SQLalchemy, supports ORMs and can accommodate different db environments (e.g. MSSQL, SQLite, etc) WOFpy architecture (frontend - webservices) Backend (WOF module) WOFpy_flask -Defines the REST methods for WaterOneFlow using Flask WOF py WOFpy – a python wrapper for WaterOneFlow Services WaterML Native metadata database -Creates URLs and views for both REST and SOAP web services. REST web service Source-specific DAOs Realtime data on web (Data Access Objects) WOFpy_soap -Defines the SOAP methods for WaterOneFlow using SOAPlib • The python libraries provide capabilities to serve REST and SOAP. Both types access the same backend. • WOFpy web services can be accessed by CUAHSI HIS clients just like any other WaterOneflow web service. SOAP web service Runserver_all.py -Management script -Launches both webservices using Werkzeug.DispatcherMiddleware WOFpy demo WOF py WOFpy – a python wrapper for WaterOneFlow Services Next steps Release WOFpy and other python modules as open-source software Developmental version on GITHub (search for WOFpy) Continue to add datasets to Add parameters to Texas Registry Conduct training workshops. Develop how-to’s for putting data into the system and also retrieving data. WDfT System will be live by end of summer WOF py WOFpy – a python wrapper for WaterOneFlow Services A Texas-specific HIS hosted by the Texas Water Development Board Employs partnerships with Federal and Texas agencies to share water data Inherits National HIS technology Provides additional tools and services to provide ease of use and a level of quality control for partners and clients (e.g. WOFpy) A fully customizable, opensource framework written in Python to publish WaterOneFlow web services User can maintain their data in their original database format Mapping to waterML is defined by user-customized Data Access Objects (DAOs) Tutorials and documentation coming soon! Acknowledgments: James Seppi WOFpy creator