From Wikipedia, the free encyclopedia Storage Resource Manager
Storage Resource Manager
The Storage Resource Management (SRM) technology Another important problem that SRMs address is
was initiated by the Scientific Data Management Group storage clogging. Storage clogging is a critical problem
at LBNL and developed in response to growing needs of for large scale shared storage systems, since the removal
managing large datasets on a variety of storage systems. of files after they are used is not automated. This in-
Dynamic storage management is essential to ensure creases the cost of storage, and slows the analysis and
(i) prevention of data loss, discovery process. SRMs help unclog temporary storage
(ii) decrease of error rates of data replication, and systems, by providing lifetime management of accessed
(iii) decrease of the analysis time by ensuring that analy- files. This capability is crucial to efficient usage of storage
sis tasks have the storage space to run to completion. under cost constraints.
There are already numerous examples where data SRMs also serve as gateways to secure data access. By
from simulations running on leadership class machines limiting external access to all storage systems through a
were lost because they were not moved in time to a mass standard SRM interface, one can assure not only authen-
storage system. Storage Resource Managers (SRMs) ad- ticated access, but also the enforcement of authorized ac-
dress such issues by coordinating storage allocation, cess to files. The SRM technology was highly successful in
streaming the data between sites, and enforcing secure SciDAC-1, and is currently used in production in several
interfaces to the storage systems (i.e. dealing with special large collaborations. SRM implementations that interop-
security requirements of each storage system at its home erate have been developed at LBNL, FNAL and TJNAF, as
institution.) For example, in a production environment, well as several sites in Europe. Furthermore, this technol-
using SRMs has reduced error rates of large-scale repli- ogy increases the scientist’s productivity by eliminating
cation from 1% to 0.02% in the STAR project. Further- the tedious and time consuming tasks of managing stor-
more, SRMs can prevent job failures. When running jobs age, performing robust data movement, and dealing with
on clusters some of the local disks get filled before the job security requirements at various storage sites.
finishes, resulting in loss of productivity, and therefore In addition to leading the SRM standard development
a delay in analysis. This occurs because space was not by coordinating with multiple institutions, the LBNL
dynamically allocated and previous unneeded files were team has developed SRM systems to disk storage and
not removed. While there are tools for dynamically al- mass storage systems, including HPSS. These SRMs have
locating compute and network resources, SRMs are the been used in several application domains, including mul-
only tool available for providing dynamic space reserva- tiple projects at the SDM center, Earth System Grid, the
tion, guaranteeing secure file availability with lifetime STAR experiment, and the Open Science Grid (OSG). As
support, and automatic garbage collection that prevents data sets continue to grow and become ever more com-
clogging of storage systems. plex, these projects depend on the continued develop-
The SRM specification has evolved into an interna- ment and support of the SRM implementations from
tional de-facto standard, and many projects have com- LBNL. It is essential to capitalize on the SciDAC-1 success-
mitted to use this technology, especially in the HEP and es and sustain current projects that depend on the SRM
HENP communities, such as the World-wide Large technology, further improving and deploying SRMs in
Hadron Collider (LHC) Computing Grid (WLCG) that sup- additional projects and application domains, and contin-
ports ATLAS and CMS. The SRM approach is to develop ued evolution of the SRM standard. Specifically, based on
a uniform standard interface that allows multiple imple- past experience, we have identified important features
mentations by various institutions to interoperate. This that require further development and coordination. Th-
approach removes the dependence on a single imple- ese include sophisticated aspects of resource monitoring
mentation, and permits multiple groups to develop SRM that can be used for performance estimation, authoriza-
systems for their specific storage resources. This ap- tion enforcement, and accounting tracking and report-
proach became crucial to the interoperation of storage ing for the purpose of enforcing quota usage in SRMs.
systems for such large scale projects that have to manage Another aspect that needs further development is SRMs
and distribute massive amounts of data efficiently and for multi-component storage systems. Such systems,
securely. Without such a unifying technology, such pro- made of a combination of multiple disk arrays, parallel
jects cannot scale, and are bound to fail. This problem file systems, and archival storage are becoming more
will only grow over time as computing facilities move in- prevalent as the volume of data that need to be managed
to the petascale regime. grow exponentially with petascale computing.
1
From Wikipedia, the free encyclopedia Storage Resource Manager
Use of SRMs in real applica- process and analyze data weeks or months sooner
than if done using only TJNAF computing resources.
tions • LBNL’s SRMs have been used in production in the
Earth Systems Grid (ESG) Project to provide
The SRM interfaces have been cooperatively defined and
transparent access from multiple remote storage
multiple implementations developed in the US and
systems at NERSC, NCAR, ORNL, LLNL, and LANL,
Europe. LBNL has introduced the concepts and subse-
including HPSS and NCAR-MSS. A disk version of an
quently led a coordinated effort of defining a
SRM has been used by the ESG portal to manage the
community-based common interface. Several implemen-
disk space when it is shared as file storage for
tations have been deployed in various applications in-
multiple clients.
cluding HEP, HENP, ESG as well as new application do-
• The use of SRMs for the CPES fusion project for
mains, such as Fusion simulation, biology, and others.
large-scale robust data movement will be
Some specifics of SRM usage to date are:
incorporated into workflow engines as part of the
• LBNL’s SRMs have been used in production over the
SDM center activities.
last few years to support intensive robust data
List of Storage Resource Manager software:
movement between BNL to NERSC at a rate of about
• APTARE StorageConsole
10,000 files (about 1 TB) per week in an automated
• BMC MAINVIEW Storage Resource Manager
fashion. This arrangement resulted in a 50X
• Computer Associates BrightStor Storage Resource
reduction in the error rates, from 1% to 0.02% in the
Manager
STAR project.
• CreekPath Acuity (Storage Resource Manager is a
• In one application, called GridCollector, SRMs were
component of Acuity)
used in combination with an efficient indexing
• DataCore Software Corporation SANmaestro Storage
method to greatly speed up the analysis of STAR. In
Resources and Performance Management and
several cases the analysis task was performed in a
Analysis
day as compared to previous efforts where scientists
• EMC ControlCenter
waited for months to sift out the relevant data. This
• BULL Calypso
work received recognition with a Best Paper Award
• Engenio Storage Performance Analyzer (OEM AppIQ
in ISC’05.
StorageAuthority Suite)
• The SRM collaboration has grown as a grass root
• HDS HiCommand Storage Services Manager (OEM
activity between LBNL, FNAL, and BNL, and later
AppIQ StorageAuthority Suite)
CERN and RAL. Consequently, a common interface
• HP SANworks
was developed, and this activity continues at this
• HP Storage Essentials (formerly AppIQ
time. This standard has been adapted by the WLCG
StorageAuthority Suite)
collaboration.
• IBM TotalStorage Productivity Center for Data
• SRMs have been used in production by several
(formerly IBM Tivoli Storage Resource Manager)
facilities including BNL, NERSC, FNAL, CERN, TJNAF,
• SGI InfiniteStorage Resource Manager (OEM AppIQ
ORNL and NCAR, and other facilities in Europe and
StorageAuthority Suite)
Asia.
• Sun StorEdge Enterprise Storage Manager (OEM
• Another example of a successful deployment is the
AppIQ StorageAuthority Suite)
SRM-dCache developed at FNAL. It is widely
• Fermilab SRM Fermilab implementation of the
deployed for use in the CMS project, and it
Storage Resource Manager interface
interoperates with the SRM-Castor at CERN. This
• Berkeley Storage Manager Berkeley Lab
effort demonstrated the usefulness of SRMs by
implementation of the Storage Resource Manager
achieving sustained SRM-to-SRM managed transfers
interface
from Castor to FNAL dCache and onto tape at a rate
• Veritas Command Central Storage from Symantec
between 40 and 60 MB/s.
• SRMs are used by TJNAF to provide the CLAS and
Lattice QCD collaborations with remote access to the See also
JASMine mass storage system. Such access has • List of SAN network management systems
allowed researchers to utilize computing resources • Storage Resource Management working group
at universities and other collaborating institutions to
Retrieved from "http://en.wikipedia.org/w/index.php?title=Storage_Resource_Manager&oldid=466418932"
Categories:
• Storage software
2
From Wikipedia, the free encyclopedia Storage Resource Manager
This page was last modified on 17 December 2011 at 23:10. Text is available under the Creative Commons Attribution-
ShareAlike License; additional terms may apply. See Terms of use for details. Wikipedia® is a registered trademark of
the Wikimedia Foundation, Inc., a non-profit organization.Contact us
Privacy policy About Wikipedia Disclaimers
3