WP 8 Data Handling And Visualization

							WP-8: Data Handling And Visualization

                      Review Meeting Report

            Felix Hupfeld, Andrei Hutanu, Andre Merzky,
                   Thorsten Schütt, Brygg Ullmer

                       Zuse-Institute-Berlin

WP-8, ZIB
            Overview

 Formal Matters
 Visualization
 Tangible Devices
 Remote File Access
 Replica Management / File
 Movement
 Metadata Service
WP-8, ZIB
            Formal Matters

 Deliverables: most right on time, some delivered
 earlier

 Milestone 8.2 (M24): Dynamic data access
 techniques are designed and implemented
        Replica catalog (Location independant file access)
        Remote File Access (Pitfalls and GridFTP)
        Visualization service

 Plan for the rest of this year:
        Further integration with other WPs
        Implement additional features (user management)


WP-8, ZIB
            Formal Matters (II)

 B. Ullmer, A. Hutanu, W. Benger, H.-C. Hege. (2003).
 Emerging Tangible Interfaces for Facilitating Collaborative
 Immersive Visualization. Workshop on Collaborative
 Virtual Reality and Visualization, October 2003.
 H.-C. Hege, A. Hutanu, R. Kähler, A. Merzky, T. Radke, E.
 Seidel, B. Ullmer. (2003). Progressive Retrieval and
 Hierarchical Visualization of Large Remote Data.
 Workshop on Adaptive Grid Middleware, September 2003.
 T. Schütt, A. Merzky, A. Hutanu, F. Schintke, Remote
 Partial File Access Using Compact Pattern Descriptions,
 IEEE CCGrid 2004.


WP-8, ZIB
               Visualization context
Grid = “virtualization” of resources
   Broad progress toward virtualization of hardware resources
   (compute power, disk space, network)
   Less common: virtualization of graphics hardware, software
“Virtualization” of visualization resources
   Scientific visualization has demanding hardware and software
   requirements, unavailable to many users
   Software is frequently complex and “heavy-weight”
   Our approach:
          Automation for core visualization operations
          Provision of grid-based “visualization services”
          Develop simplifying mobile services & devices


   WP-8, ZIB
         Visualization and mobile devices

Visualization
   • Access to grid resources
     from existing viz applications
   • Access to existing viz
     applications from the grid

Mobile devices
   • Access to viz services
     from mobile phones, PDAs
   • Development of new
     grid-enabled “mobile”
     viz devices
   WP-8, ZIB
            Visualization service

                                    1D


                                    2D




                                    3D

WP-8, ZIB
            Mobile + viz = “viz artifacts”

Motivation
   Advantages of stereo, collaborative interaction
   Importance of the meeting room, teleconferencing
   Simplifying interaction + gaining real estate
   by breaking out into the physical world




WP-8, ZIB
            Data cards




     Represent online data,
      user credentials,
      remote users,
      parameters, etc.

WP-8, ZIB
          Interaction pads
                                       upper left            upper right
                               targ
                               a uth                                             targ

                             ? se ns                                             a uth
                                                                               ? se ns
                               net
                                                                                 net



  source          ta rget                           center




                                       lower left            lower rig ht


“binding pad”                            “placement pad”
                                                                              targ
                               targ                                           a uth
                               a uth                                        ? se ns
                             ? se ns
                               net                  source                    net



  para m 1        para m 2




“parameter pad”                          “control pad”
  WP-8, ZIB
              Remote File Access (I)

   Needed for
          Visualization of remote files (intermediate results of large jobs)
          Job input files (extract and transmit only data of interest)
          …
   Existing mechanisms
          GridFTP remote partial file access (read, seek, read, seek, .. )
          I/O wrapper for local and remote access methods (to be used in
          GAT)


But does not fit Grid requirements, because
   too many small remote operations!


  WP-8, ZIB
               Remote File Access (II)

 Need to execute high-level operations on the server
  side

   File format specific approach(HDF5)
          HDF5 – structured binary file format.
          Datasets, Groups, Attributes
          Forwards HDF5 calls to the remote side
          Plugin for the Globus GridFTP server


   Generic approach (much better )  next slide


   WP-8, ZIB
            Remote File Access (III)

Generic approach
Compact access pattern description: Pitfalls




Speedup of several orders of magnitude achieved!
Example: Access part of a Cube:
(339, 65874, 262144, (0, 511, 2048, 32,
                      (0, 511, 2048, 32, (0, 3, 16, 32)))
 just 1 (in words: one!) remote call
WP-8, ZIB
      Replica Management Components

 Replica catalog
        Supports hierarchical data organization
        POSIX filesystem like interface
        Uses C++ with SOAP/GSI interface
        Adaptor inside of GAT


 File movement service
        Used for replication
        Based on GridFTP


 Adaptive service (WP6)
        Ranks replica locations
        Optimizes GridFTP parameters


WP-8, ZIB
            StorageBox Metadata Service

 Unified storage for metadata of persistent Grid objects
 (archived files, job input/output, messages/notifications)
 Any number of namespaces, configurable via pluggable access
 policies as public or private archives, group collaboration spaces or
 private annotations for public data



                           Public Sensor
                           Data Archive
            User ‘A’:                               User ‘B’:
         Sensor Data                               Sensor Data
       Annotations and                           Annotations and
      Private Messages                          Private Messages
                            Project ‘X’:
                            Collaboration
                                Space
A
                                                                         B
WP-8, ZIB
             StorageBox Metadata Service

 Portlet as user interface
 Integrated with WP12’s MessageBox service
 SOAP web service interface
 Pluggable database backends
        with PostgreSQL implementation
 Portable C++ implementation
            Win32, Linux, Mac OS X
 Planned:
        Improved query language / processing
        Remote administration interface
        More access policies



WP-8, ZIB
            Continuing plans

 Visualization, mobile devices
        Further integration with portal
        Implement additional viz operations
        Deploy, test both services & devices w/ end users
 Data management
        Provide a flexible namespace for organizing files
        Remote File Access: Generic vs. File Format
        Dependant



WP-8, ZIB
            Questions?




WP-8, ZIB
         Visualization and mobile devices

Visualization
   • Access to grid resources from existing viz applications
      Completed: remote data access from Amira
   • Access to existing viz applications from the grid
       Demoable: grid automation, presentation of reports
       integrating content from gnuplot, OpenDX, Amira;
       grid access to streaming Amira content
Mobile devices
   • Access to viz services from mobile phones, PDAs
      Completed + demoable from mobile phones, iPAQ
   • Development of new grid-enabled “mobile” viz devices
      Demoable: use of new devices to load remote content,
      activate and control video conference sessions
   WP-8, ZIB
            Videoconferencing example




   24.02.04, Berlin-Munich, surgical planning

WP-8, ZIB