LCG Middleware Testing Report
(CERN-INTAS 03-52-4297)
E.Slabospitskaya, IHEP, Russia
CERN-Russia Joint Working Group on LHC Computing
June, 25, 2006
Outline
1.Report for 2004
Testing against SRM (E. Slabospitskaya,IHEP) and GFAL
(M. Sapunov,IHEP)
II. Report for 2005
The 4 kind of tests have been developed by Russian developers:
• 1. Testing of some functionality in LCG-2 Releases (D.Zaborov, ITEP,
L.Stepanova, INR, S.Laptev, INR)
• 2. Port SFT tests to gLite
(M.Sapunov, IHEP)
• 3. Port new gLite test suites to EDG/LCG Framework
(E.Slabospitskaya, IHEP)
• 4. New functionality in gLite test suites
• (N.Klopov, PNPI)
II Report for the first part of 2006
Using the EDG/LCG Framework via NMI (L.Stepanova,INR, M.Sapunov,
IHEP)
2004
Testing against SRM and GFAL
• Storage Resource Manager (SRM) and Grid File
Access Library (GFAL) are GRID middleware
components used for transparent access to mass
storage elements
• These two Test Suites have been developed by
IHEP team:
• for testing of SRM interface v.1.1
• for testing against the GFAL file system.
• The programs have been written in C and Perl
languages
2005
Testing of new functionality in LCG-2
Releases (1/3)
• File Transfer Service (FTS) CLI based test
suite
1. A basic test of the ability of the FTS to perform successful file transfer.
2. High level cancellation test implies:
- submission of an FTS job with several files to transfer in parallel;
- waiting until at least 1 file transfer is done;
- canceling the job;
- check that files with status "Done" are kept and not deleted;
- check that files pending are canceled and not transferred.
3. FTS stress test allows to run many copies of the basic transfer test..
4. Tests of the reaction of FTS CLI commands to various "mistakes":
5. FTS management tests.
Testing of some LCG-2
functionality (2/3)
• Parsers for Gridftp and Gatekeeper
logs
Data processing is selecting from logs of gridftp and
gatekeeper. Scripts in Perl have developed for these
purposes. Results store to MySQL database for future
processing. The Perl DBI module provides a generic
interface for database access.
Testing of some LCG-2
functionality (3/3)
Parser for SFT logs
Extract old statistics from early SFT history (in PHP4)
about good/bad sites and visualization of results
( jpgraph-1.19).
- search name of region, and prepare list of regions in
html
- output image exposes the date statistic for sites into one
region (via web browser)
Port SFT tests to gLite
• Site Functional Tests (SFT)
Port the existence LGC-2 tests to gLite.
gLite SFT consist of 2 main parts: Launcher and Tests. Launcher part has
kind of GNU/Linux file structure and consist of 5 main modules. Each
module consist of submodules. Main modules called "submit", "status",
"publish", "cancel" and "clean" . Tests are using "sandbox" feature of
gLite/LCG-2 middleware. It is easy to add more tests, rules have been
defined..
Port new gLite test suites to
EDG/LCG Framework
• I/O storm testing in EDG/LCG Framework
This Framework is a set of Testing Suites produced by many
of EDG and LCG developers.
User launches multiple (parallel or one by one)
jobs from User Interface to Worker Node. Each job on Worker Node launches a
script which compiles and link C++ program and creates jdl file (this program
is a part of job in input sandbox). This C++ program via API creates a file (size
is optional) and transfers (writes/reads) these files between WN and I/O server
(there and back). The result returns to user (to User Interface).
Test creates a local file in temporary directory, copies it to I/O server, and
writes it to tmp once again
New gLite tests for the functionality
different from LCG-2
• Direct Acyclic Graph (DAG) CLI based tests
(requested by JRA1 team)
- create the JDL files for DAG with the regular structure like tree.
DAG generation is realized based on the template that defines
the Executable and JDL-attributes for three parts of the DAG
(pre-jobs, main part and post-jobs) with given number of the
levels and nodes.
- modify the created DAG structure: add new dependencies or delete
the existed ones, based on the appropriate config file. The nodes
without dependencies are removed (optional) at the end of this step.
- add new JDL attributes (requirements) for the DAG at whole or for
one level or for the definite set of nodes according to config file.
- run DAG job.
- check status of the subjobs during DAG execution.
- retrieve the nodes stdout and stderr files after DAG completion
and create the log file.
2006
• To be able to test daily grid specific
software right after the build
was chosen the NMI system. One of the
goals was to submit and execute
tests from edg-tests framework, used to test
grid middleware in CERN IT\GD group.
Next Generation Build System
• Ease of use.
• Ease of adding new components without modifying the build system
code.
• Ease of adding new platforms preferable without modifying the
build system code.
• Building multiple components at the same time if they are
independent and follow the dependencies when required..
• Log the build/test related information in the database.
• Should transfer the required externals for a particular component to
the remote build machine. This can even include externals like Java.
Flexibility on this feature is debatable.
• Framework should be smart enough to handle the operating system
signals and should report failure if any of the scripts is killed.
• Web interface for component builders/testers.
EDG/LCG tests were launched via NMI:
DPMUnitTest Tests are
successfully
ReverseDNS DavidStorm done via NMI.
MultiDStorm
GridFTP Test which
DataStorm
is done
GlobusGatekeeper CheckStorm partially via
GfalStorm NMI.
MatchMaking CopyStorm
Tests are
Sleep not touched.
UserStorm
CECycle
ProxyRenewal
JobStorm
PileStorm RMSetupTest
M.Sapunov, IHEP and L.Stapanova ,INR
Troubles with NMI system
• One of many worry concerned with Grid middleware is
using Grid proxy in self-acting software
• Absence of the presentation layer is not a good point for
NMI system.
If the job produce the special results like html or xml files
and you want to get it back, then you need to store thus
files in result.tar.gz file, download and unpack it manually.
• It's clear that NMI system is far away from production
level. Summarizing all negative points CERN IT/GD
group decide to wait 3-4 months until NMI becomes more
mature.
Main tasks in 2006-2007
• Task 1. "The development of full featured test suites
that for the certification of the new grid MW
components“
JINR (leading role), SINP-MSU, CERN-IT (consultations)
• Task 2. "The development of testing procedures
focused on special details of various software feature
PNPI (leading role), JINR, IHEP, CERN-IT (consultations)
• Task 3. ”Porting EDG-LCG framework to the gLite
environment”
IHEP (leading role), SINP-MSU , CERN-IT (consultations)