Learning Center
Plans & pricing Sign in
Sign Out



									Technical Status of the Project

            Bob Jones

                              Bob Jones –28 August 2011- n° 1

 Testbed    status

 Application   status

 Project   retreat
      Issues and actions for software process, current (EDG 1.2) and future

 Tutorials

 Summary

                                                                Bob Jones –28 August 2011- n° 2
            Testbed Status

 Application     testbed
      Running EDG 1.2 on 5 core sites (and a couple of others)
           Since first week of August (several months later than initially planned)
      Users guide and release notes available (installation guide will come later)
      Being used for application tests
      Current “show-stopper” issues found:
           Long job status problem (can’t retrieve output)
           Long file transfers problem (20 mins limit)

 Development       testbed
      Testing urgent updates to EDG 1.2
           More recent beta release of Globus 2
           Various fixes for data management chain

                                                                           Bob Jones –28 August 2011- n° 3
              Application Status
 WP8:   High Energy Physics
      LHC experiments doing tests now
      ATLAS task force

 WP9: Earth     Observation
      Installation of EDG 1.2 at ESA done
      Testing to start in September

 WP10: Biology

      Initial tests made with EDG 1.2

 Overall   comments:
      General confusion about how best to use data mgmt tools
      Software not yet stable enough and insufficient diagnostics information available
      Too difficult to configure
      Concern that EDG 1.2 in its current configuration will not scale easily to ~40 sites

                                                                               Bob Jones –28 August 2011- n° 4
               ATLAS Task Force
 Task   for with ATLAS & EDG people (lead by Oxana Smimova)

 ATLAS     is eager to use Grid tools for the Data Challenges
      ATLAS Data Challenges are already on the Grid (NorduGrid, iVDGL)
      The DC1/phase2 (to start in October) is expected to be done mostly using the Grid tools

 By   September 16 (ATLAS SW week) evaluate the usability of EDG for the DC tasks

 Thetask: to process 5 input partitions of the Dataset 2000 at the EDG Testbed +
 one non-EDG site (Karlsruhe)

 Intensiveactivity has meant they could process some partitions but problems with
 long running jobs is still an issue

 Data   Management chain is proving difficult to use and sometime unreliable
 Need   to clarify policy for distribution/installation of applications software

On-going activity with very short-timescale: highest priority task

                                                                             Bob Jones –28 August 2011- n° 5
           Project Retreat
 Project      retreat held last week (27 & 28 August) at Chevannes

 ~45     participants
        work package managers, architecture group, quality group, applications
         groups, mware experts, representatives from LCG, DataTAG, Globus &

 Agenda       and material on the web:
        Photos by Jeff Templon

3   sessions addressing most important aspects of projects current work:
        Software Release Process
        Release 1.2
        Testbed 2

                                                                     Bob Jones –28 August 2011- n° 6
               Software Process

   Over-simplification of the current situation:
    1.   Mware groups develop software in isolation
    2.   ITeam assembles it as best it can
    3.   Site managers are asked to install it
    4.   Application groups are asked to test it


    No place for the mware groups to integrate software before delivering it
    to the ITeam
    Inadequate software testing – leads to
    installation/configuration/execution faults
    We are running blind – no way to control or reliably plan software delivery

                                                                Bob Jones –28 August 2011- n° 7
               Software Process: Autobuild

A release manager will be nominated with overall responsibility for ensuring
 the procedure is followed
 Makeautobuild tools the basis of the daily work of the mware groups and
        Nightly build from CVS repository for all software
              Problems must be fixed ASAP – checked by Quality Group reps
        Mware groups give ITeam CVS tags instead of RPMs
              Tagged software must be documented
        Mware group must perform and supply unit tests
              Integrated with nightly build

       software that fails the integration, testing or is inadequately
 Tagged
 documented will be rejected
        Mware group is responsible for fixing it

                                                                             Bob Jones –28 August 2011- n° 8
        Software Process: Quality Group

 Recently formed Quality Group, convened by Gabriel Zaquine, is
 responsible for ensuring quality issues are addressed within the WPs
     Ensure unit test plans are complete and followed
     Follow-up on problems reported bugzilla & nightly builds
     Organise running of code checking tools on all EDG software
     Agree on adopted project developer-guidelines etc.


                                                                 Bob Jones –28 August 2011- n° 9
         Software Process: Testing

 Strengthen     the Testing Group
      Identify leader and a small number of full-time testers
      Assemble and maintain test suite integrated with autobuild tools

 Automate     installation and configuration of software releases
      To permit auto testing need to be able to auto install & configure a release on a
       pre-defined small example site
      Needs improvements by mware WPs to simplify and complete installation &
       configuration of their sw
      Site managers have good overview about how to do this
         Need to clarify the work involved during this conference

 Set-up   certificate testbed
      Used for testing activities
      Involves several sites

                                                                          Bob Jones –28 August 2011- n° 10
                 Technical Management
 Architecture        group documenting testbed 2 architecture
      draft:
      Meets once a month (next meeting tomorrow)

 Project     Tech. Board addresses deliverables and relationships with other projects
      Meets once per quarter (next meeting 2 nd October @ CERN)

 Need       more frequent technical management forum
      Authority to make technical & architectural decisions affecting sw development in WPs
      Include WP managers, chaired by the Technical Coordinator
             Can call on mware experts according to needs of the themed agenda
      Meets frequently to ensure issues are addressed rapidly
             Associate with WP managers weekly meeting

 Relationship      with Architecture Group and Project Tech. Board needs to be clarified

                                                                                  Bob Jones –28 August 2011- n° 11
            Testbed Support

 Strengthen user        support group
      Ensure people involved have sufficient knowledge of the software
      Emphasis on the accurate and usefulness of the responses provided
           Tools used for support are a secondary issue
      Federate with equivalent groups from other projects
      Provides support on the application testbed

 Clarify    & document procedures
      Creating a new CA (CA group)
           Need to reduce time involved (currently 3 months)
      Site Installation (site managers & ITeam)
           Steps for system manager and requirements for a site to join the testbed
      Creating & Managing a Virtual Organisation (site managers & ITeam)
           Steps involved and tasks of a VO manager

                                                                        Bob Jones –28 August 2011- n° 12
            Release Development
                        Current “show-stoppers” fixed

              1.2                  Continuous support

  CVS                                                                         Migrate sites
Autobuild               branch      patches         Application tests
 Nightly                                            satisfied
                          2.0    Incremental Improvements         Continuous Support

      Changes foreseen for 1.3 & 1.4
      become “incremental improvements”
                                                            2.x       Incremental Improvemen


                                                                      Bob Jones –28 August 2011- n° 13
                 Incremental Steps from EDG 1.2

       1.   Fix “show-stoppers” for application      7.    Giggle & Reptor – WP2
            groups – mware WPs (continuous)
                                                     8.    LCAS with dynamic plug-in modules –
       2.   Build EDG1.2.x with autobuild tools -          WP4
                                                     9.    NetworkCost Function – WP7
       3.   Integrate testing framework and
            limited automatic tests with autobuild   10.   Integrate mapcentre (nordugrid?) and
            tools - testing group                          R-GMA – WP3
       4.   Automatic installation & configuration
            procedure for pre-defined site (can’t    11.   GLUE modified info
            auto test without it)                          providers/consumers – WP1,4,5

       5.   Start autobuild server for RH 7.2 and    12.   Res. Broker – WP1
 End        attempt build of release 1.2 – Yannick
Sept        Patois                                   13.   LCFG for RH 7.2 – WP4
       6.   New LCFG - WP4                           14. Integration with Condor as batch
                                                         system – WP4
       7.   GridFTP server access to MSS - WP5                         Expect this list to be
            What do we do about: Space mgmt, VOMS, slashrgrid?                this week

                                                                                   Bob Jones –28 August 2011- n° 14
        EDG Tutorial
The tutorials are aimed at users wishing to "gridify" their applications
using EDG software and are organized over 2 full consecutive days
user:griduser passwd:tutorials123

DAY1                                     DAY2
 Tutorial   introduction
                                          Data   Management
 Introductionto Grid computing and
                                          LCFG, fabric mgmt & sw
 overview of the DataGrid project
                                          distribution & installation
 Security
                                          Applications   and Use cases
 Testbed    overview
                                          Future   Directions
 Job   Submission
                                          hands-on   exercises: data mgmt
 hands-on   exercises: job submission

                                                                  Bob Jones –28 August 2011- n° 15
                   Tutorial rehearsal
 Rehearsal        at CERN, 29 & 30 August
        19 participants (members of project or closely related) to check material & approach

 Lessons       learnt
        Can’t cover as much material as we hoped (goes to fast)
              Explain why not just how
              Avoid details – can read them from references afterwards
        Need as many helpers as possible for hands-on exercises
        Participants have difficulties with certificate management
              All participants must have a certificate ready for them and be in the same VO

 Generated        a lot of enthusiasm in the participants and EDG people doing the hands-on
        Found genuine bugs during hands-on exercises
        Recommend mware WPs send developers to help with hands-on exercises
        New project people should follow the tutorial

   Thanks to:
           Mario Reale, Elisabetta Ronchieri, Akos Frohner, Erwin Laure, Peter Kunszt, Antony Wilson, Steve
            Fisher, Maite Barroso Lopez, Owen Synge, Emanuele Leonardi, Steve Traylen, Frank Bonnassieux,
            Christophe Jacquet, Sophie Nicoud, Karin Burghauser & CERN training people
                                                                                          Bob Jones –28 August 2011- n° 16
                  Tutorial Schedule
 CERN        school of Computing, Naples, 23-27 September
       80 participants. Hands-on exercises only (presentations by Carl Kesselman & Ian Foster)
       ALL EDG people attending should do exercises first and help others at the school

 CERN,       October 3 & 4
 NeSC,       Edinburgh, December
       Dates still moving. Maximum 30 participants (more for the presentations)

 We    could accommodate more sites in December, January etc.
       Sites must provide support and handle logistics
             Organisers/helpers must attend tutorial at another site first

 The   tutorial does represent some load on the testbed (own VO & cert. creation)
 For   the future
       Hands-on exercises are a test suite - automate and run with the nightly checks
       The material must be kept up to date with each public release of the software
             We need to nominate people responsible for the different chapters of the tutorial to be responsible
              for ensuring the slides and exercises are kept up to date

                                                                                         Bob Jones –28 August 2011- n° 17
 Addressing the serious bugs found by the application groups on the testbed
 is the task with the highest priority

 Testing   activities need more resources
 Test-bed    support is becoming a more important task

 Future   releases must continue to address the needs of the application groups

 We   need to clarify the following points during this conference:
      Autobuild status & how automate installation & configuration
      Contents of the test-suites
      Release plans until the next EU review

In short:
        What we are doing is right, we are just going about it in a sloppy manner
        Need to go one step at a time and ensure each step works

                                                                      Bob Jones –28 August 2011- n° 18

To top