Docstoc

PowerPoint Presentation - CUAHSI HIS

Document Sample
PowerPoint Presentation - CUAHSI HIS Powered By Docstoc
					Using Python to implement
a Hydrologic Information
System for Texas
                           James Seppi
              (CRWR, University of Texas at Austin)

                     Dharhas Pothina
                    (Texas Water Development Board)

                                 Ernest To
                             (Espey Consultants, Inc)

                         Tim Whiteaker
              (CRWR, University of Texas at Austin)

                        Andrew Wilson
                    (Texas Water Development Board)
Texas Water Development Board

   Provides leadership and scientific expertise for
    development and conservation of water
   Accumulate and publish large-scale State water
    databases in a form accessible as WaterOneFlow web
    services.
   Create a Texas-centric HIS Water Data for Texas:
     Partnerships with Federal and Texas agencies to
      obtain data
     Inherits National HIS technology
     Adds Texas-specific services to
      provide ease of use and a level of
      quality control
Why a Texas-specific HIS?

   National HIS technology available but uptake low among
    Texas agencies
       lack of motivation
       lack of time/resources
       lack of experience with the technology

   National HIS doesn’t quite do everything TWDB needs.
       Parameter mapping
       Unit Conversions
       Web Viewer
         original paradigm

           Put your data in an
             ODM database




i.e. learn swahili
this didn’t get much traction…

   Data is usually already in the format the
    organization needs.
   Changing formats/delivery mechanisms is a
    pain.
   ODM DB Schema doesn’t always fit the needs of
    data collecting agencies.
   Resources. Resources. Resources.

Overall response: yawn
new paradigm… make it easy

   Form partnerships to obtain data
   Provide technical know-how and support
   Provide multiple paths to serve data
   Provide a useful product with branding that is
    available to the general public can use.
   Scheduled updates to datasets


     i.e. we have trained swahili translators
           available please speak to us
   Four Components of


2. Translation
   services to                                1. Existing WaterML
   convert non-                                  compliant web
   WaterML data                                  services within
   sources into                                  Texas
   WaterML




                                               4. Access Tools for
                                                   data discovery
                        WaterML                    and download:
3. WDFT Central         compliant
   Registry service     web
     Limited
                        services
      Parameter Set
     List of Texas
      HIS data
      sources                                           HydroExcel,
                                                         HydroDesktop,
     Unit
                                    GEMSS                HydroGET…
      conversions
                                    (TNRIS hosted map based
                                    web interface)
Why python?
   Built-in cross-platform compatibility
   Availability of a massive variety of scientific
    libraries/modules
   Amazing community support. Better than
    commercial vendors
   $FREE!$
Python @TWDB

   pINT - Python INstrument Toolkit
   SWIS - Surface Water Information System
   WOFpy - Python WaterOneFlow Wrapper
   pyHIS - Python HIS Client
   pyHAT - Python Hydrographic Analysis Toolkit
   pyselfe/pug – hydrodynamic model analysis &
    visualization
          WOFpy concept

                                 WOF py




WOFpy – a python wrapper for WaterOneFlow Services
Old paradigm (i.e. ODM db as intermediate)
                                Data in ODM




                                                                    WaterML
  Native
 database




       ODDataloader                                    Generic
       SSIS scripts                                    ODwebservices
       SQL views
                              •Significant learning curve
       Manual loading, etc.
                              •Data persists in ODM under Microsoft SQL server
                               i.e., not in native format and environment
                              •“When is the next update?”
                              •“What if data is collected in real-time?”
Old paradigm (i.e. ODM db as intermediate)
                                 Metadata in ODM




                                                                  WaterML
                                                                  for GetSites,
                                                                  GetVariables,
  Native
                                                                        etc.
 metadata




                                                        Generic
       Loading scripts                                  ODwebservices


  Real-                                                            WaterML
  time                                                             for GetValues
 data on
  web
                         Customized webscraper web service
Old paradigm(i.e. single framework)
New paradigm (ODM db as intermediate)
                                    Metadata in ODM




                    Single framework                                                                   WaterML
                    • Dynamically map between native database             for GetSites,
                       schema and WaterML                                 GetVariables,
 Native
metadata            • Allow data to persist in native database                etc.
                    • Accomodate different database environments
                       (MSSQL, SQLite, etc)                               WaterML
                    • Handle both static and real-time data
                    • Dispatch and manage WaterOneFlow services
                       in both SOAP                            WOF py
                                                               Generic
                       and
        Loading scripts REST                                   ODwebservices
                    • Open-source and fully
                       customizable
                                                  WOFpy – a python wrapper for WaterOneFlow Services

  Real-                                                                                                WaterML
  time                                                                                                 for GetValues
 data on
  web
                           Customized webscraper web service
 WOFpy architecture (backend – data access)
                                                                      WOF py
Data persists                                                                             (WOF module)
in source’s
native db                            WOFpy – a python wrapper for WaterOneFlow Services




environment     • WOF module assembles
                  WaterML elements to WaterML
                  responses.
                • Implements a reduced ODM
                  data model (‘Model’) for
                  mapping to WaterML elements.
  Native        • Model is also mapped to the
 database
 metadata         native db of the source by
                  source-specific DAOs.
                                                                                                         WaterML



                Source-specific
                    DAOs
                  (Data Access                                      • Model contains classes
                                                                       DAOs define object-relational mapping (ORM)
                                                                      common to both
                                                                       from native schema to Model.
   Real-            Objects)                                          WaterML and ODM. translating user
                                                                    • Contains methods for
   time                                                                arguments into queries to native db.
  data on                                                           • The python module, SQLalchemy, supports
                                                                       ORMs and can accommodate different db
   web                                                                 environments (e.g. MSSQL, SQLite, etc)
WOFpy architecture (frontend - webservices)


                                                                           WOFpy_flask
                                                                           -Defines the REST
Backend (WOF module)                                                         methods for
                                                                            WaterOneFlow                           REST
                                             WOF py
                                                                             using Flask                            web
                                                                           -Creates URLs and                      service
                                                                 WaterML
  Native
            WOFpy – a python wrapper for WaterOneFlow Services

                                                                             views for both
 database
 metadata
                                                                            REST and SOAP
                                                                              web services.
             Source-specific
                 DAOs
                (Data Access
   Real-          Objects)
   time
  data on
   web
                                                                           WOFpy_soap
                                                                                                                 SOAP
                                                                           -Defines the SOAP
                                                                              methods for                         web
                                                                             WaterOneFlow                        service
                                                                             using SOAPlib

• The python libraries provide capabilities
  to serve REST and SOAP. Both types
  access the same backend.                                                                 Runserver_all.py
• WOFpy web services can be accessed                                                  -Management script
  by CUAHSI HIS clients just like any                                                 -Launches both webservices using
  other WaterOneflow web service.                                                     Werkzeug.DispatcherMiddleware
                WOFpy demo

                                 WOF py




WOFpy – a python wrapper for WaterOneFlow Services
Next steps
    Release WOFpy and other python modules as
     open-source software
        Developmental version on GITHub (search for
         WOFpy)

    Continue to add datasets to
    Add parameters to Texas Registry
    Conduct training workshops.
    Develop how-to’s for putting data into the system
     and also retrieving data. WDfT System will be live
     by end of summer
                                                                              WOF py




                                             WOFpy – a python wrapper for WaterOneFlow Services


   A Texas-specific HIS hosted by           A fully customizable, open-
    the Texas Water Development               source framework written in
    Board                                     Python to publish
   Employs partnerships with                 WaterOneFlow web services
    Federal and Texas agencies to            User can maintain their data in
    share water data                          their original database format
   Inherits National HIS technology         Mapping to waterML is defined
   Provides additional tools and             by user-customized Data
    services to provide ease of use           Access Objects (DAOs)
    and a level of quality control for       Tutorials and documentation
    partners and clients                      coming soon!
                                             Acknowledgments:
    (e.g. WOFpy)                             James Seppi
                                             WOFpy creator

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:4
posted:2/6/2012
language:
pages:18