Open Source SDMX software by hcj

VIEWS: 29 PAGES: 18

									                                    23 April 2011
                                Eurostat Luxembourg




    SDMX Open Source Software

Erik van Ingen
FAO/D4Science




                      D4Science-II project   |   www.d4science.eu
                                                                             Content


 • OpenSDMX is used in
    • FAO/TechCDR
    • FAO/Fisheries (www.fao.org/figis/sdmx)
    • EU/D4Science (http://opensdmxdevelopers.wikispaces.com/PisaCNR2011January)
 • Looking at ECB & FED
 • Why OpenSDMX?
 • OpenSDMX pillars
 • OpenSDMX Development Environment & Technical Architecture
 • OpenSDMX Communication channels
 • Synergy analyses MT, Eurostat & OS
 • JAXB and XMLBeans



                                                                                        2
SDMX Open Source Software
SDMX open source software                                                   www.d4science.eu
23 April 2011
                             TechCDR :Corporate Data Repository



 TechCDR is a platform of services
    Provide external access to FAO digital assets
         Search and access statistics, maps, feature layers, documents,
          imagery, etc.
         External APIs
     Provide internal services to Analytical Field Systems
         Workspace management, ETL uploads, security, publishing,
          notification and update alerts




                                                                           3
SDMX open source software                                      www.d4science.eu
                                                               TechCDR Platform
                                                                                                                             front-end layer

                                                           Portlet container
                                               Portlets, Javascript, Flex components

       Working              Metadata                                                                  Ontologies          Custom
       System               Manager                                                                    Manager          Applications

                                                                                                                                service layer

    Working System                                                                           Business                     Integration
       Services                                                                              Services                       Services

                                                                  JBoss ESB

     Content                                       Infrastructural                        Data Analysis                      Statistical
   Management        ETL Jobs       Metadata           Services                                               SDMX
                                                                        GIS Services        Services                           Data
     Services        (Talend OS)    Services        (Security, log,      (Geoservers)
                                                                                                             Services
     (Alfresco)                                       cache, etc)
                                                                                             (Pentaho)                        Services


                                                                                                                                 data layer
                                    Staging Area
      Working           Content                                                         Statistical
                                           Structured                 Metadata                            Multidimensional
      System           Repository                                                         Data
                                         Data Repository              Repository                            Data Model
      RDBMS                                                                             Repository             (DWH)
                                                                                                                                                4
SDMX open source software                                                                                                www.d4science.eu
                               TechCDR use of OpenSDMX



 TechCDR publishes multidimensional data and artifacts in
   SDMX
 OpenSDMX features:
    OpenSDMX repository services obtain datasets from OLAP
     database
    OpenSDMX services dynamically map data in OLAP cubes
     to SDMX datasets
    Codelists, Codes, DSDs are published




                                                             5
SDMX open source software                        www.d4science.eu
                                     TechCDR SDMX Services



   OpenSDMX services exposed via
      Restful API following SDMX 2.1 guidelines
   TechCDR Custom
      Interfaces to OpenSDMX services




                                                              6
SDMX open source software                         www.d4science.eu
                                                                      TechCDR SDMX Services




                                               REST
                                                                         front-end layer
                                             Controller

                                              SDMX
                                             Services
                                                                         service layer
                                                          Olap4j
                                Relational
                                  Data                    XMLA
                                 Services
                                                    Data Analysis
                                                      Services
                                                          (Pentaho)




                                     Statistical      Multidimensional
                   Metadata
                                       Data             Data Model
                   Repository
                                     Repository            (DWH)           data layer




                                                                                                       7
SDMX open source software                                                                  www.d4science.eu
                            OpenSDMX and D4Science



 Codelist
 uploader




                                                      8
SDMX open source software                 www.d4science.eu
                                                        Looking at the
                                the ECB SDMX visualisation framework:


 Federal Reserve Bank of New York and European Central Bank


 Why Open Source:                         Logistics and License
 •Rapid, robust development               •ECB, NY FED, BOC
 •Reduced institutional costs             •Code hosted and publicly available on
 •More thorough code review               Google Code
 •Better code support                     •Strict commit policy and change
                                          management procedures
                                          •BSD license
                                          •Roles
                                          •Leadership roles (Project Manager,
                                          Project Advisor and Project Architect)
                                          •Committers: three individuals
                                          •Contributors: contributions are welcome
                                          from all
                                          •Users: the code is available to all

                                                                                  9
SDMX open source software                                             www.d4science.eu
                                           Why OpenSDMX?



 • A platform was needed in order to serve three different
   projects
 • The platform needed to be open source in a full fledged
   manner.
 • FAO and D4Science needed to have a good governance
   position in the open source project.




                                                                10
SDMX Open Source Software
SDMX open source software                           www.d4science.eu
23 April 2011
                                         OpenSDMX pillars



 • Implementation of version 2.1 REST webservice
   specification
 • Supporting (only) version 2.1 (still migrating yet from
   version 2.0)
 • Focus on core SDMX Artefacts (Codelist, DSD, Generic and
   Compact Dataset)
 • Integratable in every context through adapters
 • Not necessarily having its own persistence layer, maybe
   added in the future




                                                              11
SDMX Open Source Software
SDMX open source software                         www.d4science.eu
23 April 2011
                            OpenSDMX Development Environment



 •   Building with Maven
 •   Continuous Integration with Continuum
 •   Code Metrics with Sonar & FindBugs
 •   Artifact server with Artifactory
 •   SCM with Subversion on SourceForge
 •   Documentation on Wikispaces
     (http://opensdmxdevelopers.wikispaces.com/)
 •   Eclipse IDE mostly used amongst developers
 •   Automatic deployment on Tomcat
 •   Integration testing
 •   Unit Testing

                                                                12
SDMX open source software                           www.d4science.eu
                            OpenSDMX Technical Architecture



 •   Spring for dependency injection, REST and caching
 •   JAXB as the Java binding for XML
 •   EHCache for caching
 •   Java 6




                                                                 13
SDMX open source software                            www.d4science.eu
                            OpenSDMX Communication channels




 https://sourceforge.net/p/opensdmx/home/
 http://opensdmx.wikispaces.com/
 http://opensdmxdevelopers.wikispaces.com/
                                                               14
SDMX open source software                          www.d4science.eu
                             Synergy analyses MT, Eurostat & OS


 Synergy Level               MT-OS   MT-ES   ES-OS   Mean in %
 Java 6
 Development Environment
  *Maven
  *Continuous Integration
  *Unit Testing
  *Integration Testing
  *Code Metrics
 Technical Architecture
  *Spring
  *Caching
  *XML Binding
  *Persistence
  *Layering and interfaces
 Functional Architecture
 Open Source Organised
 License                                                               15
SDMX open source software                                  www.d4science.eu
                                        JAXB and XMLBeans I



 Metadata Technology uses XMLBeans, OpenSDMX uses
   JAXB. Eurostat uses a proprietary XML binding.

 XMLBeans
 • Email traffic from 18/02/2011-18/03/2012 is 15 (10
   developermail, 10 usermail)
 • Last release 2.5.0 was on 14-12-2009

 JAXB
 • Email traffic from 18/02/2011-18/03/2012 is 70 usermail
 • Last release 2.2.3.1 was on 16/01/2011
                                                                    16
SDMX open source software                               www.d4science.eu
                                     JAXB and XMLBeans II



 Observations:
 • Spring supports Castor, Jaxb, Jibx and XmlBeans
 • XmlBeans is less active than JAXB
 • JAXB is natively part of Java




                                                                 17
SDMX open source software                            www.d4science.eu
                                     END




               End of Presentation




                 Erik van Ingen
                 FAO/D4Science




                                                       18
SDMX Open Source Software                  www.d4science.eu
23 April 2011

								
To top