Workflow Template Generation

Document Sample
Workflow Template Generation Powered By Docstoc
					Workflow Template
   Generation
      Haw to
                       Steps                   Hand Made
                                               Preparation

• Make proper set of WF examples on Portal
• Save it to a local directory
• Decompress it
• Make a Template Directory of it (TD)
• Define the Table of Actual Parameters TAP
• Apply the Macro Generator to gain Destination Directory
  MG( TD,TAP) -> DD
• Apply the Directory Compress And Upload Program
• Start the Submit All Command on the Portal
What should be parameter?
   Formal Parameter             Name                   Explanation
          wf            Source Workflow name
        wf_dest         Name of workflow to       Name must be defined
                        be generated.             in accordance with the
                                                  automatic visualization
                                                  system
      vo<jobseq>        Virtual organization of   <jobseq> is integer in
                            the selected job          the range 1..3
           se              Storage Element
                           destination host
       r<jobseq>         direct resource with     <jobseq> is integer in
                          jobmanager of the           the range 1..3
                             selected job
   f<jobseq><portseq>    Logical name of the      <jobseq> is integer in
                         eventual remote file         the range 1..3
                           belonging to the       <portseq> is integer in
                         selected port of the         the range 1..2
                             selected job
       e<jobseq>         List of environment          It is just a string
                           variables of the       following the syntax in
                             selected job                  the JDL
  Where are these Parameter?
• <wf>_files.wrk -> <wf_dest>_files.wrk
  <wf_dest>,<ri>,<fi,j>
• <JOB>.grid
  <voi>
• .<JOB>.desc
  <voi>,<ri>
• <JOB>/<JOB>.jdl
   <voi>,<ri>,<fi,j>,<ei>
                                   Resource                                              RemoteFile

                                 <wf>_files.wrk
workflow "<<wf_dest>>"
{"MultipLOC" 2 "MultipRemLoc" 1 OFFLINE                  "Copyrem" 1 "MultipLOC" 1 OFFLINE}
{"Copyrem" SEQ_PORTAL (is_instrumented=false;monitor=off)
"/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/Copyrem/CopyAndTime"
{""}"LINUX"
{"<<r1>>"}
              0 "/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/Copyrem/0/INPUT"
(file_type=local;original_path="D:\A-TEST\I1") PERMANENT INPUT                    1 "<<f11>>"
(file_type=remote;local_path="/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/OUTPUT")
PERMANENT OUTPUT"MultipLOC" SEQ_PORTAL (is_instrumented=false;monitor=off)
"/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/MultipLOC/multiply.exe"
{""}"LINUX"
{"<<r2>>"}
              0 "/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/MultipLOC/0/INPUT1"
(file_type=local;original_path="D:\A-TEST\I1") PERMANENT INPUT                    1 "<<f21>>"
(file_type=remote;local_path="/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/INPUT2")
PERMANENT INPUT              2 "/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/MultipLOC/2/OUTPUT"
(file_type=local;original_path="OUTPUT") PERMANENT OUTPUT "MultipRemLoc" SEQ_PORTAL
(is_instrumented=false;monitor=off)
"/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/MultipRemLoc/multiply.exe"
{""}"LINUX"
{"<<r3>>"}                                                                                                       New FW
              0 "<<f31>>"
(file_type=remote;local_path="/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/INPUT1") name
PERMANENT INPUT              1
"/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/MultipRemLoc/1/INPUT2"
(file_type=local;original_path="") PERMANENT INPUT                    2
"/home/krisztian/pgportal/portal_work/users/hermann/<<wf_dest>>_files/MultipRemLoc/2/OUTPUT"
(file_type=local;original_path="OUTPUT") PERMANENT OUTPUT}
<JOB>.grid



  <<vo1>>
                    <JOB>.desc




hermann Indirect_REMOTE_Template Copyrem <<vo1>> <<r1>> false
    <JOB>/<JOB>.jdl
[   VirtualOrganisation = "<<vo1>>";
    Executable = "Copyrem.sh";

    JobType = "Normal";
    StdOutput = "std.out";
    StdError = "std.err";
    InputSandbox = {"CopyAndTime",
                      "Copyrem.sh","0/INPUT",
                     "info.tar.gz"      };
    OutputSandbox = {"std.out", "std.err", "std.log"};
    OutputData = { [ OutputFile = "OUTPUT";
                      LogicalFileName = "<<f11>>“
                    ] };
    Environment = {<<e1>>};
     RetryCount = 3;
     Requirements = other.GlueCEInfoHostname == "<<r1>>";
]
                 Table of Actual Parameters
wf               wf_dest                             se              vo1            r1                               f11             f12              e1
Indirect_REMO                                                        seegrid_LCG_                                                                     "L
      TE_Templ                                                             2_BROK   gw01.rogrid.pub.ro:/jobmana      lfn:/grid/seegrid/hermann/Test
      ate        T_CO_SEQ_REM                                              ER            ger-lcgpbs-seegrid                  _2006_03_29_17_00
aaa              bbb                                 ccc             ddd            eee                              fff             ggg              hh
Indirect_REMO                                                        seegrid_LCG_                                                                     "L
      TE_Templ                                                             2_BROK   gw01.rogrid.pub.ro:/jobmana      lfn:/grid/seegrid/hermann/Test
      ate        Indirect_REMOTE_Template                                  ER            ger-lcgpbs-seegrid                  _2006_03_29_17_00
                                                                     seegrid_LCG_
                                                                           2_BROK   gw01.rogrid.pub.ro:/jobmana      lfn:/grid/seegrid/hermann/Test
U_B_C__N_Q       U_B_C__N_Q                                                ER            ger-lcgpbs-seegrid                  _2006_03_29_17_00        "L
                                                                     seegrid_LCG_
                                                                           2_BROK   grid2.cs.bilkent.edu.tr:/jobma   lfn:/grid/seegrid/hermann/Test
U_B_C__N_Q       U_B_C_grid2.cs.bilkent.edu.tr_N_Q                         ER             nager-lcgpbs-seegrid               _2006_03_29_17_00        "L
                                                     SE_SUBSTI       seegrid_LCG_
                                                         TUTI              2_BROK   grid2.cs.bilkent.edu.tr:/jobma   lfn:/grid/seegrid/hermann/Test
U_B_C__S__Q      U_B_C__S__Q                             ON                ER             nager-lcgpbs-seegrid               _2006_03_29_17_00        "L
                                                     grid1.cs.bilk   seegrid_LCG_
                 U_B_C_grid2.cs.bilkent.edu.tr_S_g         ent.ed          2_BROK   grid2.cs.bilkent.edu.tr:/jobma   lfn:/grid/seegrid/hermann/Test
U_B_C__S__Q          rid1.cs.bilkent.edu.tr _Q             u.tr            ER             nager-lcgpbs-seegrid               _2006_03_29_17_00        "L
     Haw To Generate Tab ?
1. Fill the Excell Table
2. When Save as, select
   Type : Text(Tab delimited)
Haw to call the Macro Generator ?
 public WorkflowMacroFactory(String templateDirPath,
                             String tapDirPath,
                             String destinationDir,
                             boolean toUNIX)

 A test main program calls this by default parameters:

 new WorkflowMacroFactory(
  "d:\\A_A_WorkflowRepository\\TEMPLATES\\DecompressedTemplates",
  "d:\\A_A_WorkflowRepository\\TEMPLATES\\TAP_newComplex.txt",
  "d:\\A_A_WorkflowRepository\\TOM",
  false);

 External call:

 Java – Gener.jar <TD> <TAP> <DD>
Apply the Directory Compress And
         Upload Program
• Under construction
• Temporary problems:
  - Zip –Unzip sensitive to OS
  - Upload should bypass WF Editor,
    because SAVE operation of WE
    overwrites <JOB>/<JOB>.jdl files