Learning Center
Plans & pricing Sign in
Sign Out

VLe SP2.5-Nov-8


									Realize an e-Science workflow bus:
technical issues and agenda

   Zhiming Zhao
   VL-e SP2.5 weekly meeting

• Some recent events
• Workflow bus
  –   Concept and challenges
  –   Typical scenario
  –   An agent based design
  –   Agenda
Recent events: Krakow Grid workshop
• Background
  – Since 2001
  – Organized by the Polish Grid community: AGH, Cyfronet,
  – Invited talks, oral and posters
  – Different tracks: security, performance, applications,
• Our work
  – Z. Zhao, D. Vasunin, A. Wibisono et al., “Privacy issues in
    integrating R environment in scientific workflows”: oral
  – In the Grid security track, and paper submission 1/Dec.
• Contacts
  – Met some Grid security experts, e.g., Syed Naqvi (CCLRC
    Rutherford Appleton Laboratory, UK), in the PC of WSES 07.
  – Interesting talk
      • From EU commission
      • From different projects: EGEE, KF-Grid, D-Grid,
Cont. the 20th CODATA Int’l Conf.

• Background
   –   CODATA: Int’l org on Data for Science and Technology
   –   20th meeting, in Beijing
   –   Invited talk, abstracts and posters
   –   Keynote talk: Toney Hey
• Our talk
   – On behalf of Bob: “Scientific Workflow management in
     Virtual Laboratory for e-Science (VL-e)”
   – Workflow support in VL-e
• Contacts
   – Prof. Yan Baoping, director of e-Science project in Chinese
     academy of sciences
        • Interested in research cooperation
   – Prof. Huang Lican, P2P service discovery
An e-Science workflow bus

                                    Scientific experiment:
                                      a meta workflow

      Sub                       Sub                          Sub                     Sub
   workflow 1                workflow 2                   workflow 3              workflow 4

      Taverna                 Triana                     Kepler               VLAMG

                            Workflow bus: provide services for
                1) Interoperability and integration, 2) composition, 3) provenance,
                          4) Enactment, 5) Human in the loop computing
                                Generic Grid middleware
Research lines

• Integration and interoperability
• Composition support for meta workflows
• Experiment provenance via a workflow bus
• Workflow enactment and different computing
• Interactive workflow execution and human in the
  loop computing
Research lines

• Prototype a basic workflow bus: Integration
  and interoperability
• Composition support for meta workflow
• Experiment provenance via workflow bus
• Workflow enactment and different computing
• Interactive workflow execution and human in the
  loop computing
  The basic idea

 WF 1             WF 2                Workflow
Scenario       Scenario             Study              Control         Composition          Other e-Sci.
manager        manager             manager            interface         interface             services

                           Workflow runtime infrastructure

 • Decompose intelligence, e.g., integration, coordination,
   monitoring and runtime control, into scenario managers,
   study manager and user interfaces.
 • Distributed components are loosely coupled via a runtime
    –   Z. Zhao et. al., VLWF-BUS: a workflow bus for multi e-Science domains, IEEE Int’l Conf. e-
        Science and Grid computing.
A basic set of components: for the first
•   Workflow runtime infrastructure (WF-RTI):
     – Coupling distributed components
     – Handling data distribution: messages, data objects and files
     – Interfacing other e-Science services: semantic tools, Data base,
•   Scenario manager:
     – Wrapping legacy workflow engines
     – Coordinating the execution of the sub-workflow
•   Study manager
     – Executing meta workflows
     – Orchestrating scenario managers, and providing runtime control
•   Control interface
     – Monitoring workflow execution
     – Controlling runtime behavior
•   Composition interface
     – Composing meta workflow
     – Providing tools for (semi) automatic composition
     – Validating the meta workflow
     A typical runtime scenario

Composition   Control       Study         Scenario          Scenario

      Meta WF
                  Meta WF                                              Meta WF
                                Data location

                                                Data location
                                                Data location
     Cont.                                                             - How does a study
                                                                       manager find a proper
                                                                       workflow engine and assign
                                                                       a scenario manager?
Composition   Control       Study         Scenario          Scenario
                                                                       - How does a study
                                                                       manager manage the
      Meta WF                                                          application boundary (talk
                  Meta WF                                              to correct scenario manager
                                    WF1                                in a correct workflow)?
                                                                       - How does a scenario
                                Data location
                                                                       manager decide where to
                                                                       store results of the sub
                                                     WF2               workflow?
                                                                       - How will a study manager
                                                Data location
                                                                       be fault tolerant: when sub
                                                Data location          workflow failed, or in
                                                                       possible deadlock
                                                                       - How will scenario
                                                                       managers and study
                                                                       manager handle privacy
                                                                       issues and X display?
Design considerations
• Middleware for message passing and data
   – Standardized interface
• The integration paradigm between components
   – C/S, Federated
   – Flexible
• Control intelligence realization
   – Customized implementation or use AI based
Agent based design

• Scenario manager, study manager and user
  interface are encapsulated as agents
   – Functionality of legacy workflow engine and behavior
     of scenario manager are explicitly described
   – Study manager implements intelligences for
     scheduling and orchestrating scenario managers
   – Study manager is able to talk to generic Grid
     services and to enact (execution planning) workflows
• A multi agent framework is used as workflow
  runtime infrastructure (WF-RTI)
   – Runtime information of the agents and workflows are
     distributed via the agent framework
   – The agent framework provides feasible infrastructure
     for realizing provenance and monitoring functionality
What is an agent?
• Agent is an intelligent
  component:                                             World
   – Behavior:                                           model
       • sensors

       • Effectors                                     Capability
       • Internal activities
   – Brain:                                              Control
       • World model                                   intelligence
           – Identity
           – Task description
       • Capability                                     Internal
           – Constraints of activity                    activities
       • Intelligence
           – Activity scheduling mechanism
           – Interaction strategies
  Design paradigm

                 Brain         World model         Task          Work
                                                 description     flow
                Sensor           base           Observations

                Effector       Capability


Scenario         Study
manager         manager

 Kepler    Taverna       Triana
SceMnger   SceMnger     SceMnger
Scenario manager: sub-workflow
engine wrapper
Wraps sub-workflow engine and provide uniform interface for Study manager
• World model:
     – Profile: ID (unique), name,
     – workflow description (content, or handle),
     – Brain (World model of the entire system, execution state of the sub-
•   Capability
     – Control operations for workflow engine
          • wf_read/wf_clearn, wf_start/wf_pause/wf_stop/wf_restart, wf_save/wf_restore,
     – Engine control operation
          • eg_load/eg_kill/eg_migrate
     – Scenario manager/ study manager:
          • scen_start, scen_stop, scen_migrate
     – Scenario manager/ Scenario manager:
     – query state, inform data location,
•   Intelligence
     – Be able to start engine, and migrate it at runtime
     – Be able to detect the runtime state of the other scenarios and pass data
Study manager: Workflow
Coordinates the execution of sub-workflows
• Word model
     – Profile (ID, Type, Name, Location)
     – Workflow (description)
     – Brain:
          • Execution plan (schedule plan, execution model)
          • Knowledge backbone
          • World model
•   Capability
     – Study manager/ composition: execute_workflow
     – Study manager/ control interface: query_state,
     – Study manager/scenario manger:
          • workflow execution,
          • scenario manager control,
          • Sub-workflow execution,
•   Intelligence
     – Execute a workflow
     – Coordinate sub-workflows
Decoupled interface

• Composition agent: compose workflow and
  activate the study manager

• Runtime control agent: check the runtime
Implementation agenda

•   What do we have now
     – Suresh’s work: integration study
     – Conceptual study on workflow interoperability
•   Version 1: basic prototype
     – Taverna and Kepler will be chosen as test case
     – Study manager, scenario manager and user interface will be
     – April/2007: demonstrate the prototype, performance study,
       technique report (or publication).
•   Based on Version 1, the system will be improved incrementally
     – Version 2: implement application use case
         • Ketan’s work
         • IBU Marco Roos’s case
     – Version 3: with composition support
         • Results from KF-Grid
     – Version 4: data provenance
         • Cooperation with Ikay, Victor
     – Version 5: flexible execution

To top