VLe SP2.5-Nov-8

					Realize an e-Science workflow bus:
technical issues and agenda

   Zhiming Zhao
   VL-e SP2.5 weekly meeting
Outline

• Some recent events
• Workflow bus
  –   Concept and challenges
  –   Typical scenario
  –   An agent based design
  –   Agenda
Recent events: Krakow Grid workshop
• Background
  – Since 2001
  – Organized by the Polish Grid community: AGH, Cyfronet,
    etc.,
  – Invited talks, oral and posters
  – Different tracks: security, performance, applications,
• Our work
  – Z. Zhao, D. Vasunin, A. Wibisono et al., “Privacy issues in
    integrating R environment in scientific workflows”: oral
    presentation
  – In the Grid security track, and paper submission 1/Dec.
• Contacts
  – Met some Grid security experts, e.g., Syed Naqvi (CCLRC
    Rutherford Appleton Laboratory, UK), in the PC of WSES 07.
  – Interesting talk
      • From EU commission
      • From different projects: EGEE, KF-Grid, D-Grid,
Cont. the 20th CODATA Int’l Conf.

• Background
   –   CODATA: Int’l org on Data for Science and Technology
   –   20th meeting, in Beijing
   –   Invited talk, abstracts and posters
   –   Keynote talk: Toney Hey
• Our talk
   – On behalf of Bob: “Scientific Workflow management in
     Virtual Laboratory for e-Science (VL-e)”
   – Workflow support in VL-e
• Contacts
   – Prof. Yan Baoping, director of e-Science project in Chinese
     academy of sciences
        • Interested in research cooperation
   – Prof. Huang Lican, P2P service discovery
An e-Science workflow bus

                                    Scientific experiment:
                                      a meta workflow

      Sub                       Sub                          Sub                     Sub
   workflow 1                workflow 2                   workflow 3              workflow 4




      Taverna                 Triana                     Kepler               VLAMG




                            Workflow bus: provide services for
                1) Interoperability and integration, 2) composition, 3) provenance,
                          4) Enactment, 5) Human in the loop computing
                                Generic Grid middleware
Research lines

• Integration and interoperability
• Composition support for meta workflows
• Experiment provenance via a workflow bus
• Workflow enactment and different computing
  models
• Interactive workflow execution and human in the
  loop computing
Research lines

• Prototype a basic workflow bus: Integration
  and interoperability
• Composition support for meta workflow
• Experiment provenance via workflow bus
• Workflow enactment and different computing
  models
• Interactive workflow execution and human in the
  loop computing
  The basic idea


                                       Meta
 WF 1             WF 2                Workflow
Scenario       Scenario             Study              Control         Composition          Other e-Sci.
manager        manager             manager            interface         interface             services


                           Workflow runtime infrastructure

 • Decompose intelligence, e.g., integration, coordination,
   monitoring and runtime control, into scenario managers,
   study manager and user interfaces.
 • Distributed components are loosely coupled via a runtime
   infrastructure
    –   Z. Zhao et. al., VLWF-BUS: a workflow bus for multi e-Science domains, IEEE Int’l Conf. e-
        Science and Grid computing.
A basic set of components: for the first
prototype
•   Workflow runtime infrastructure (WF-RTI):
     – Coupling distributed components
     – Handling data distribution: messages, data objects and files
     – Interfacing other e-Science services: semantic tools, Data base,
       etc.
•   Scenario manager:
     – Wrapping legacy workflow engines
     – Coordinating the execution of the sub-workflow
•   Study manager
     – Executing meta workflows
     – Orchestrating scenario managers, and providing runtime control
•   Control interface
     – Monitoring workflow execution
     – Controlling runtime behavior
•   Composition interface
     – Composing meta workflow
     – Providing tools for (semi) automatic composition
     – Validating the meta workflow
     A typical runtime scenario

Composition   Control       Study         Scenario          Scenario


      Meta WF
                  Meta WF                                              Meta WF
                                    WF1
                                                                       WF
                                                                       1
                                Data location

                                                     WF2
                                                Data location
                                                                        WF
                                                                        2
                                                Data location
                                                                       Issues:
     Cont.                                                             - How does a study
                                                                       manager find a proper
                                                                       workflow engine and assign
                                                                       a scenario manager?
Composition   Control       Study         Scenario          Scenario
                                                                       - How does a study
                                                                       manager manage the
      Meta WF                                                          application boundary (talk
                  Meta WF                                              to correct scenario manager
                                    WF1                                in a correct workflow)?
                                                                       - How does a scenario
                                Data location
                                                                       manager decide where to
                                                                       store results of the sub
                                                     WF2               workflow?
                                                                       - How will a study manager
                                                Data location
                                                                       be fault tolerant: when sub
                                                Data location          workflow failed, or in
                                                                       possible deadlock
                                                                       situations?
                                                                       - How will scenario
                                                                       managers and study
                                                                       manager handle privacy
                                                                       issues and X display?
Design considerations
• Middleware for message passing and data
  distribution
   – CORBA/HLA, MPI/SOAP or FIPA
   – Standardized interface
• The integration paradigm between components
   – C/S, Federated
   – Flexible
• Control intelligence realization
   – Customized implementation or use AI based
     framework
Agent based design

• Scenario manager, study manager and user
  interface are encapsulated as agents
   – Functionality of legacy workflow engine and behavior
     of scenario manager are explicitly described
   – Study manager implements intelligences for
     scheduling and orchestrating scenario managers
   – Study manager is able to talk to generic Grid
     services and to enact (execution planning) workflows
• A multi agent framework is used as workflow
  runtime infrastructure (WF-RTI)
   – Runtime information of the agents and workflows are
     distributed via the agent framework
   – The agent framework provides feasible infrastructure
     for realizing provenance and monitoring functionality
What is an agent?
                                                         Brain
• Agent is an intelligent
  component:                                             World
   – Behavior:                                           model
       • sensors




                                                                      Effectors
                                             Sensors
       • Effectors                                     Capability
       • Internal activities
   – Brain:                                              Control
       • World model                                   intelligence
           – Identity
           – Task description
       • Capability                                     Internal
           – Constraints of activity                    activities
       • Intelligence
           – Activity scheduling mechanism
           – Interaction strategies
  Design paradigm
                                                Identification

                 Brain         World model         Task          Work
 Agent
                                                 description     flow
                               Knowledge
                Sensor           base           Observations

                Effector       Capability

                  Gui
                                       Description

Scenario         Study
manager         manager



 Kepler    Taverna       Triana
SceMnger   SceMnger     SceMnger
Scenario manager: sub-workflow
engine wrapper
Wraps sub-workflow engine and provide uniform interface for Study manager
• World model:
     – Profile: ID (unique), name,
     – workflow description (content, or handle),
     – Brain (World model of the entire system, execution state of the sub-
       workflow)
•   Capability
     – Control operations for workflow engine
          • wf_read/wf_clearn, wf_start/wf_pause/wf_stop/wf_restart, wf_save/wf_restore,
            data_input/data_output
     – Engine control operation
          • eg_load/eg_kill/eg_migrate
     – Scenario manager/ study manager:
          • scen_start, scen_stop, scen_migrate
     – Scenario manager/ Scenario manager:
     – query state, inform data location,
•   Intelligence
     – Be able to start engine, and migrate it at runtime
     – Be able to detect the runtime state of the other scenarios and pass data
Study manager: Workflow
coordinator
Coordinates the execution of sub-workflows
• Word model
     – Profile (ID, Type, Name, Location)
     – Workflow (description)
     – Brain:
          • Execution plan (schedule plan, execution model)
          • Knowledge backbone
          • World model
•   Capability
     – Study manager/ composition: execute_workflow
     – Study manager/ control interface: query_state,
     – Study manager/scenario manger:
          • workflow execution,
          • scenario manager control,
          • Sub-workflow execution,
•   Intelligence
     – Execute a workflow
     – Coordinate sub-workflows
Decoupled interface

• Composition agent: compose workflow and
  activate the study manager

• Runtime control agent: check the runtime
  information
Implementation agenda

•   What do we have now
     – Suresh’s work: integration study
     – Conceptual study on workflow interoperability
•   Version 1: basic prototype
     – Taverna and Kepler will be chosen as test case
     – Study manager, scenario manager and user interface will be
       integrated
     – April/2007: demonstrate the prototype, performance study,
       technique report (or publication).
•   Based on Version 1, the system will be improved incrementally
     – Version 2: implement application use case
         • Ketan’s work
         • IBU Marco Roos’s case
     – Version 3: with composition support
         • Results from KF-Grid
     – Version 4: data provenance
         • Cooperation with Ikay, Victor
     – Version 5: flexible execution

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:12/20/2011
language:
pages:19