Docstoc

Time Series Plot from the Data Portal - CUAHSI-HIS

Document Sample
Time Series Plot from the Data Portal - CUAHSI-HIS Powered By Docstoc
					How to share and publish your
       data using HIS

       David G Tarboton
        Jeff Horsburgh
         Ilya Zaslavsky
        Tom Whitenack
        David Valentine

                                 Support
         http://his.cuahsi.org   EAR 0622374
                         Outline
• CUAHSI HIS data publication system         Tarboton  5 min
• Observations data Model (ODM)              Tarboton  20 min
• Using ODM                                  Horsburgh 40 min
    • SQL Management Studio
    • ODM Tools
    • ODM Data Loader
    • Streaming Data Loader
    • Controlled vocabulary editing
    • SSIS for data loading                  Tarboton    10 min
• HIS Central
    • Data Publication Workflow              Zaslavsky   10 min
    • WaterML and configuring WaterML        Valentine   15 min
      web services over ODM
    • Registration of your data service in   Whitenack 20 min
      HIS central
                                     HIS Data Publication System
Query, Visualize, and Edit
 data using ODM Tools                         Analysis               Access            Discovery
                                                                                        Hydroseek
                                           GIS                         HydroExcel
                                          Matlab       Java
                                                       C++              HydroGet
                                          Splus                        HydroLink
                                            R           VB
                                                                      HydroObjects
                                           IDL




                                                                       Service Registry        Hydrotagger
                                                   GetSites
                  Streaming       ODM              GetSiteInfo
                     Data                          GetVariableInfo
                   Loader        Database          GetValues
   Base Station                                      WaterML           Harvester
   Computer(s)
                       ODM                         WaterOneFlow
     Telemetry         Data                         Web Service
     Network
                      Loader

                                                                                      Water Metadata
                   Excel                                                  ODM
                              Text                                                       Catalog
                                          Contribute                       ODM
                                          your ODM
 Sensors
                                                                                     HIS Central
         Steps in publishing data
1. Server setup or access
2. Store observations in ODM
3. Provide access to data through web services
  (http://<your-server>/<your-network>/cuahsi_1_0.asmx?WSDL)

4. Index the resulting water data service at HIS
   Central (http://hiscentral.cuahsi.org)
 Why an Observations Data Model

• Syntactic heterogeneity (File types and formats)
• Semantic heterogeneity
  – Language for observation attributes (structural)
  – Language to encode observation attribute values
    (contextual)
• Publishing and sharing research data
• Metadata to facilitate unambiguous
  interpretation
• Enhance analysis capability
                   Scope
• Focus on Hydrologic Observations made at a
  point
• Exclude Remote sensing or grid data. These
  are part of a digital watershed but not suitable
  for an atomic database model and individual
  value queries
• Primarily store raw observations and simple
  derived information to get data into its most
  usable form.
• Limit inclusion of extensively synthesized
  information and model outputs at this stage.
    What are the basic attributes to be
associated with each single data value and
    how can these best be organized?

Value        Offset                Method
DateTime     OffsetType/           Quality Control
Variable     Reference Point       Level
Location     Source/Organization   Sample Medium
Units        Censoring             Value Type
Interval     Data Qualifying       Data Type
(support)    Comments
Accuracy
       Point Observations Information Model
Utah State     Data Source
   Univ
          Little Bear River    Network                                        GetSites

                                                                            GetSiteInfo
       Little Bear River at Mendon Rd Sites
                                                                          GetVariableInfo
                         Dissolved Oxygen Variables
                                                                          GetValues
                   9.78 mg/L, 1 October 2007, 6PM Values
                                                             {Value, Time, Qualifier, Offset}
  •   A data source operates and provides data to an observation network
  •   A network is a set of observation sites (stored in a single ODM instance)
  •   A site is a point location where one or more variables are measured
  •   A variable is a measured property (e.g. describing the flow or quality of water)
  •   A value is an observation of a variable at a particular time
  •   A qualifier is a symbol that provides additional information about the value
  •   An offset allows specification of measurements at various depths in water
  Horsburgh, J. S., D. G. Tarboton, D. R. Maidment and I. Zaslavsky, (2008), "A Relational Model for
  Environmental and Water Resources Data," Water Resour. Res., 44: W05406, doi:10.1029/2007WR006392.
 CUAHSI Observations Data Model
• A relational database at the            Streamflow        Groundwater
  single observation level                                     levels
  (atomic model)
• Stores observation data
                               Precipitation
  made at points                & Climate                         Soil
• Metadata for unambiguous                                       moisture
                                                                  data
  interpretation
• Traceable heritage from raw
  measurements to usable               Water Quality          Flux tower
  information                                        Time
                                                                 data
• Standard format for data
  sharing
• Cross dimension retrieval
  and analysis                                              Space

                                             Variables
CUAHSI Observations Data Model




http://his.cuahsi.org/odmdatabases.html
                         Site Attributes




SiteCode, e.g. NWIS:10109000
SiteName, e.g. Logan River Near Logan, UT
Latitude, Longitude Geographic coordinates of site
LatLongDatum Spatial reference system of latitude and longitude
Elevation_m Elevation of the site
VerticalDatum Datum of the site elevation
Local X, Local Y Local coordinates of site
LocalProjection Spatial reference system of local coordinates
PosAccuracy_m Positional Accuracy
State, e.g. Utah
County, e.g. Cache
   Independent of, but can be coupled to
        Geographic Representation
   ODM                   Arc Hydro
                                                                                                          Feature
Observations Data Model
                                                                                                                                                                        (
                                                                                                                                                                        !
                                                                                                      Waterbody
                                                                                                                                                   (
                                                                                                                                                                   Watershed
                                                                                                                                                   !
                                                                HydroPoint
  Sites                                                  1                                            HydroID                                                      HydroID
                  1
                                                                                                               (
                                                                                                               !            (
                                                                                                                            !         (
                                                                                                                                      !         (
                                                                                                                                                !          !(
                                                                                                                                                            (
                                                                                                                                                            !
                                                                                                                                                            !
                                                                                                                                                           ((
                                                              HydroID
                                                                                                                                                            !                (
                                                                                                                                                                             !
  SiteID                                                                                          (
                                                                                                  !   HydroCode                               ( (!
                                                                                                                                              ! !                  HydroCode
                                                              HydroCode
                                                                                                                                                 (
  SiteCode                                                                          *                 FType                                                        DrainID
  SiteName                                                    FType                                       ((
                                                                                                          !!                                                                         (
                                                                                                      Name
                                                                                                                                                                                     !
                                                              Name                                                                                                 AreaSqKm (
  Latitude
                                                                                                                                                                             !
  Longitude
                                 OR                           JunctionID
                                                                                              (
                                                                                              !       AreaSqKm                                                     JunctionID
                                                                                                      JunctionID
                                                                                                                                ((            (
                                                                                                                                                                   NextDownID
                                                                                                                                !!            !
  …                                                                                               (
                                                                                                  !                    ((                 (
                                                                                                                                          !                                 (
                                                                                                                                                                            !
                                                                             (
                                                                             !                                         !!                                  (
                                                                                                                                                           !
                                                                    (
                                                                    !       (
                                                                            !                             (
                                                                                                          !   *
                                                                                                                                               (
                                                                                                                                               !                                 *
                                                        ComplexEdgeFeature                                                  SimpleJunctionFeature
                            CouplingTable                                                                                                                               (
                                                                                                                                                                        !
                          1 SiteID                                                            ((
                                                                                              !!
                            HydroID
                                                                                (
                                                                                !
                                            1
                                                                (
                                                                !                       (
                                                                                        !
                                                              HydroEdge             (                          (
                                                                                                               !                                                    (
                                                                                                                                                                    !
                                                                                    !
                                                                                                                                     HydroJunction
                                                                        (                                                                          (
                                                                                                                                                   !
                                                                        !
                                                             HydroID                                                        1             (
                                                                                                                                          !                          1
                                                             HydroCode                                                               HydroID
                                                             ReachCode                                                               HydroCode
                                                                            (
                                                                            !                                                             (
                                                                                                                                          !   (
                                                                                                                                              !                (
                                                                                                                                                               !
                                                             Name
                                                             ( (
                                                             ! !                                                                     NextDownID
                                                             LengthKm                                                                LengthDown            (
                                                                                                                                                           !
                                                                                            HydroNetwork                             DrainArea
                                                             LengthDown
                                                             FlowDir                                                                 FType                                               (
                                                                                                                                     Enabled
                                                                                                                                                                                         !
                                                             FType
                                                    (
                                                    !                                                 (
                                                                                                      !
                                                         ( (
                                                         ! !        (
                                                                    !                                                                                              ((
                                                                                                                                                                   !!
                                                             EdgeType                                                                AncillaryRole     (
                                                                                                                                                       !
                                                (
                                                !        ( (
                                                          ! !Enabled            ( ( (( (
                                                                                ! ! !! !                                    (
                                                                                                                            !
                                                                                                                                                                            (
                                                                                                                                                                            !
                                                                EdgeType                                           (
                                                                                                                   !
                                                                                                          (
                                                                                                          !
                                                                Flowline

                                                                Shoreline
Variable attributes
                                     Cubic meters per second
                                     Flow
                                     m3/s




  VariableName, e.g. discharge
  VariableCode, e.g. NWIS:0060
  Speciation, e.g. N in “Nitrogen, NH4 as N”
  SampleMedium, e.g. water
  ValueType, e.g. field observation, laboratory sample
  IsRegular, e.g. Yes for regular or No for intermittent
  TimeSupport (averaging interval for observation)
  DataType, e.g. Continuous, Instantaneous, Categorical
  GeneralCategory, e.g. Climate, Water Quality
  NoDataValue, e.g. -9999
            Scale issues in the interpretation
                        of data
                                        The scale triplet
           a) Extent                                  b) Spacing                                 c) Support
quantity




                                           quantity




                                                                                      quantity
               length or time                             length or time                              length or time


 From: Grayson, R. and G. Blöschl, ed. (2000), Spatial Patterns in Catchment Hydrology: Observations and Modelling, Cambridge
 University Press, Cambridge, 432 p, http://www.catchment.crc.org.au/special_publications1.html.
           The effect of sampling for measurement scales not commensurate with
                                      the process scale

                                              1.25      (a) spacing too large – noise (aliasing)

                                              0.75




                                              0.25




                                              -0.25




                                              -0.75




                                              -1.25
                                  1.25




                                  0.75




                                  0.25




                                 -0.25




                                              (b) extent too small – trend
                                 -0.75




                                     1.5
                                 -1.25



                                         1




                                     0.5




                                         0




                                    -0.5




                                         -1




                                    -1.5                  (c) support too large – smoothing out

From: Grayson, R. and G. Blöschl, ed. (2000), Spatial Patterns in Catchment Hydrology: Observations and Modelling, Cambridge
University Press, Cambridge, 432 p, http://www.catchment.crc.org.au/special_publications1.html.
Discharge, Stage, Concentration and Daily
            Average Example
                     Data Types
•   Continuous (Frequent sampling - fine spacing)
•   Sporadic (Spot sampling - coarse spacing)
                              t
•   Cumulative       V( t )   Q( )d
                                                 t  t
•   Incremental               0
                                        V (t )   Q( )d
                                V (t )
•   Average          Q (t )                        t
                                 t
•   Maximum
•   Minimum
•   Constant over Interval
•   Categorical
  Offset


OffsetValue
Distance from a datum or
control point at which an
observation was made
OffsetType defines the type of
offset, e.g. distance below
water level, distance above
ground surface, or distance
from bank of river
Water Chemistry from a profile in a lake
Groups and Derived From Associations
Stage and Streamflow Example
 Daily Average Discharge Example
Daily Average Discharge Derived from 15 Minute Discharge Data
ValueAccuracy
A numeric value that quantifies measurement accuracy
defined as the nearness of a measurement to the
standard or true value. This may be quantified as an
average or root mean square error relative to the true
value. Since the true value is not known this may should
be estimated based on knowledge of the method and
measurement instrument. Accuracy is distinct from
precision which quantifies reproducibility, but does not
refer to the standard or true value.
            ValueAccuracy




 Accurate           Low Accuracy        Low Accuracy,
                                         but precise
        Data Quality




Qualifier Code and Description provides qualifying information
about the observations, e.g. Estimated, Provisional, Derived,
Holding time for analysis exceeded
QualityControlLevelCode records the level of quality control
that the data has been subjected to.
- 0. Raw Data
- 0.5 Partly quality controlled
- 1. Quality Controlled Data
- 2. Derived Products
- 3. Interpreted Products
- 4. Knowledge Products
Series of Observations



A “Data Series” is a set of all the observations
of a particular variable at a site.

The SeriesCatalog is programmatically
generated to provide users with the ability to
do data discovery (i.e. what data is available
and where) without formulating complex
queries or hitting the DataValues table which
can get very large.
               Loading data into ODM
                                             OD Data Loader
• Interactive OD Data Loader (OD
  Loader)
   – Loads data from spreadsheets and
     comma separated tables in simple
     format                                         SDL
• Scheduled Data Loader (SDL)
   – Loads data from datalogger files on a
     prescribed schedule.
   – Interactive configuration
• SQL Server Integration Services
  (SSIS)
   – Microsoft application accompanying              SSIS
     SQL Server useful for programming
     complex loading or data
     management functions

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:2/28/2012
language:
pages:26