Collection of Statements

ETH Board Swiss National Strategic Plan for High Performance Computing and Networking (SNSP-HPCN) 2008-11 Appendix G Collection of Statements of Swiss HPCN Stakeholders at the Open Day in Berne December 6, 2006 July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-1 ETH Board In preparation of a structured discussion at the Open Day of December 6, 2006, Swiss stakeholders in HPCN were invited to provide a written statement on eight specific questions, taking into account the four boundary conditions set in the mandate of the ETH Board to the SNSP-HPCN Project Group. In order to address a broad group of stakeholders, individual members of the Project Group invited a large number of potentially interested people who were in turn given the opportunity to suggest further names. Fifty-one stakeholders actively participated in the process, either by sending a written statement or presenting their view at the meeting, or both. For reference the generic text of the letter that was sent to the stakeholders is given below: October 31, 2006 Swiss National Strategic Plan for High Performance Computing and Networking, SNSP-HPCN Dear I am contacting you on behalf of the Vice President of Academic Affairs of EPFL, Giorgio Margaritondo. He was charged by the ETH Board to chair the Project Group that must develop a National Strategy for High Performance Computing and Networking in Switzerland for the period 2008-2011. The Project Group will host an OPEN DAY on December 6, 2006 (13:15 – 17:00) in Bern, Wildhainweg 21 (“Plenarsaal 21” of the Swiss National Science Foundation). There it will collect input from the various stakeholders regarding HPCN in Switzerland. As a key person involved in supercomputing and networking I like to kindly invite you to participate in this meeting. Before going into further details of the meeting let me briefly summarize how the project was initiated, specify the aim of the project, and provide you with essential information about the project organization. Project initiation - Recently, the Swiss National Supercomputing Center (CSCS) was evaluated. This evaluation was considered as a first step towards developing a Swiss strategy for HPCN, a mandate given to the ETH Board by State Secretary for Education and Research, Charles Kleiber, in May 2006. A project organization was drafted and approved by the ETH Board on September 27/28, 2006 (see enclosures). The kick-off meeting of the Switzerland-based members of the Project Group took place October 25, 2006. The next step will be to collect the essential information from a broad group of Swiss stakeholders with a strong interest in HPCN, in particular during the OPEN DAY of December 6, 2006 in Bern. Aim of the project - The objective is to create optimal conditions for the continuation of a leading role of the Swiss National Supercomputing Center (CSCS), to harmonize the different Swiss initiatives and partners relevant to this domain (such as SIB and SWITCH), to achieve maximum return from the investments, to meet the needs of Swiss universities, industries and research institutions and to support the future international competitiveness of Switzerland in science, technology and industrial development. This coordinated effort is essential because of the accelerating technological developments in HPCN (e.g., the recent deployment of the first petaflop-level system) and of its impact not only on science and technology but also on security, risk management and key Swiss industrial sectors such as July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-2 ETH Board banking/insurance, pharma/health/biotech, engineering and nano/micro manufacturing. The project must deliver an intermediary report by January 31, 2007, and a coordinated strategic plan by July 2007, based on a global analysis of the Swiss needs within a European context and on contributions from top-level international HPCN experts. Project organization - The project will be elaborated by the coordinated work of three different bodies under a project organization as decided by the ETH Board: • The Steering Committee, • The Project Group, • The Reference Group. The members of the Steering Committee are Professor Alexander Zehnder, ETH Board (chair), Professor Ernst Hafen (President ETH Zurich) and Professor Piero Martinoli (President USI). Its responsibility is to evaluate the recommendations and the intermediate and final reports elaborated by the Project Group and to approve the final report for submission to the ETH-Board and to the political authorities. The Project Group is chaired by Professor Giorgio Margaritondo (Vice President for Academic Affairs at EPFL) and includes Professor Dimos Poulikakos (Vice President for Research, ETH Zurich), Professor Ralph Eichler (Director PSI), Dr. Marie-Christine Sawley (Director CSCS), Professor Ernest Feytmans (Director of SIB), Dr. Fiorenzo Scaroni (SWITCH), Professor Manuel Peitsch (Head of global Systems Biology, Novartis), Professor Ursula Roethlisberger (EPFL), Professor Ivo Sbalzarini (ETH Zurich), Professor Andrew Jackson (ETH Zurich), Dr. Kurt Baltensperger (ETH Board, executive secretary), plus the international members Professor Roberto Car (Princeton Univ.), Professor Knut Faegri (University of Oslo), and Professor Anwar Osseyran (Director HPC Center of the Netherlands). The Reference Group includes experts of HPCN from Switzerland and abroad and will provide for the Project Group the necessary feedback on strategic options and on specific technical issues, constituting the counterpart for debating the solutions proposed by the Project Group. It will be elected by the Steering Board and be consulted by the Project Group as required for the elaboration of its proposals. Meeting details - At the OPEN DAY on December 6, 2006, eight key questions shall be discussed by the various stakeholders in Swiss HPCN (see below). I am asking you to contribute your view on the questions that are in your interest with a brief written statement (500 words maximum; in English if possible). Please send your statement back to me as your contact person and in copy to the executive secretary of the Project Group, K. Baltensperger (kurt.baltensperger@ethrat.ch). Since the time frame is rather tight, we should receive your input by December 1, 2006. At the OPEN DAY you will have the opportunity to briefly present your view to the audience (summarized on a single overhead). Your written statement will provide the basis for the intermediate and final reports. Your written statement and your oral comments should address the following key questions; please comment on those that apply to your case, presenting the general needs of your domain of activity or industrial sector: July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-3 ETH Board 1. Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the top-performing computing system that Switzerland should acquire in 2008-2011? 2. What is the optimal role of CSCS within the overall strategy and the ideal balance between such a role and other actions? 3. What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy – that should be considered while planning the acquisition and deployment of new HPCN facilities? 4. What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? 5. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc…)? 7. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? 8. How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? Please consider the following boundary conditions agreed upon by the ETH Board, when drafting your statement: 1. Considering the clearly demonstrated positive achievement record of CSCS and the human and financial resources invested by Switzerland for its development in the Ticino Canton, this center will continue in its role as a leading element in the Swiss HPCN strategy. Such a role should be enhanced – most notably by strengthening the CSCS academic ties – and not diminished. 2. The funding projections must realistically take into account the general financial situation of Switzerland and in particular the planned public investments in science, technology and industrial development. 3. The Swiss strategy must be harmonized as much as possible with the foreseen developments in Europe and must avoid unjustified duplications. Particular attention must be devoted to the developing new strategy of the European Commission for large research infrastructures. However, the Swiss national security and industrial competitiveness should be considered as top priorities. 4. The Project Group must consider in its proposals the role of the SWITCH Network, of the several existing GRID initiatives and of the Swiss Institute of Bioinformatics (SIB). I truly hope that you will be able to attend the meeting. However, even if you cannot participate in person, please send your written statement: your opinion is very important for our objectives! May I ask you to confirm your attendance or - in case of your inability to attend – to suggest another person that is representing your field of expertise? I am expecting your answer by the end of November. Thank you for you participation. You will receive further details for the meeting at the end of November. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-4 ETH Board Sincerely, Enclosures: - Project Organization SNSP-HPCN - Decision of the ETH Board of Sept. 27/28, 2006 July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-5 ETH Board PAUL SCHERRER INSTITUT Memorandum Datum: Von: Telefon: Raum: E-mail: 1. Dezember 2006 A. Adelmann 4233 WLGA / U125 Andreas.Adelmann@psi.ch An: cc: R. Eichler K. Baltensberger (ETH-Rat) Response and Comments to the Questionnaire on SNSP-HPCN First I will clarify a few of the technical terms appearing in the questionnaire, following general comments to the HPCN complex. The petaflop-level system is a term which requires clarification: in particular one has to distinguish between the many variants of performance measures: a) theoretical peak performance b) LINPACK (Rmax ) measure used in the TOP500 and c) sustained application performance. The HPC-community will enter the area of Petascale computing most probably in late 2008 and will be based on performance measures a) and b). From the user point of view, a more subtle and appropriate definition, and one which will be used throughout this note, is: A Petascale Computing System that is both usable and useful for Switzerland is a staged HPC system that delivers application performance (measure c) above 1 Petaflop/sec. The System reaches the performance when a substantial fraction of the TOP500 is above 1 Petaflop/sec (using measure c). At NERSC (National Energy Research Centre, Berkley USA) 1 sustained Petaflop/sec is estimated for 2010, with 150’000 - 500’000 (multi-core) processors consuming in the order of x ∗ 10 MW power, with 1.2 ≤ x ≤ 5. On the cost side, the NSF (National Science Foundation, USA) released a bid for 200 M$, investment costs for a Petascale Computing System, not including building and electricity infrastructure. Three applications are selected for the bidding process namely: Molecular dynamics, turbulence flow simulations (DNS, structured girds 3D FFT) and 4D QCD (3 space dimensions and dimension in time). Scientific HPCN efforts, in its next dimension, will be a collaborative team science approach, where a single PI (principal investigator) can not drive a project. A famous example is climate modelling community, which already developed large community codes. On the funding side the Scientific Discovery through Advanced Computing (SciDAC) program in the USA is one example of an effort to fund and hence build up collaborate HPCN knowledge (code/frameworks etc.). The following must be seen in the light of the previous statements. Response to Question 1: Switzerland must acquire a Petascale Computing System. This is of utmost importance for the future of many important scientific disciplines. The Petascale Computing System (as defined above) is by 1 private communication H. Simon, associate laboratory director for computing sciences at Berkeley Lab July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-6 ETH Board definition a staged HPC system. The main reasons for a staged system are as follows: i) the HPCcommunity (at large) is not prepared to use a Petaflop-Machine as defined by in measure c), ii) with a staged system we can leverage the cost and invest in other evenly important areas such as: algorithms, data management and education. This is an important step towards the efficient use of the upcoming HPCN resources. Response to Question 2: I use the term ’Ecosystem for HPC’ as defined in 2 . Platforms, software, institutions, HPC-applications and people who solve problems involving supercomputing can be thought of collectively as an ecosystem. CSCS should be stimulating and to some extend hosting such an ecosystem. Given the fact that Petascale Computing Systems are very resource demanding (investment and operational) we can only succeed if such a system is perfectly embedded and matched to the Swiss research and development landscape, i.e. in a functional HPCN-ecosystem. The existing/traditional HPCN-ecosystem will, in the area of Petaflop computing, augmented with a third dimension besides the HPCN-Center and the user. The third dimension is the Vendor which must be included evenly. This because of the increased complexity of hardware, system software and the resulting knowledge transfer to the researchers. Response to Question 3: Fact: the data-volume will grow proportionally with the computing power. This basically states that most scientists depend on weak-scaling of their problems (eg. when they run at higher concurrencies, they are typically running proportionally larger problems). The consequence is that we are confronted with enormous amounts of data 3 . Data post-processing (data mining and visualisation) must be addressed with the same priority as the race for FLOPS (floating point operations). Response to Question 4: We have to create HPC-research capabilities which take care of basic education needs and drive the academic, interdisciplinary contribution in the field of HCPN. Much emphasis must be given to develop multidisciplinary science in the area of HPCN. Given the history of Swiss-HPC engagements an independent HPCN-Institute could stimulate new development. In minimum a research group which is embedded within the CSCS/ETH organisation must be established. Response to Question 5,6 and 8: CSCS must be a user lab sharing the resources with the Swiss academic community and the industry. The current status within the ETH-Domain must be re-evaluated. We should adapt the successful userlab-model implemented at PSI in the context of particle accelerators. In this model large research instruments are shared among different institutions. A super computer today and even more tomorrow must be seen as a large scientific instrument which shares the intrinsic challenges in administration and usage than for example a particle accelerator. Response to Question 7: HPCN is the third leg of science. With Petascale Computing Systems we will enter into a new area. 2 3 See NRC (National Research Council) - Report on ’The Future of Supercomputing’ Jim Gray’s recent paper provides some general guidance MS TR 99 100 Rules of Thumb in Data Engineering.pdf at http://research.microsoft.com/ gray/papers and there are the golden rules defined by the ASCI (Accelerated Strategic Computing Initiative) program http://www.ornl.gov/sci/optical/docs/Tutorial19991108Nowak.pdf Seite 2 page G-7 PAUL SCHERRER INSTITUT, CH-5232 Villigen PSI, Schweiz July 4, 2007 SNSP-HPCN | Final Report | Appendix G ETH Board HPCN will be a science enabler in areas ranging from basic science to applied science including security relevant areas such as: weather and disaster forecast, cryptography and simulations of complex financial, ecological and technical systems (particle accelerators, fusion etc. ). Many fields have just started to explore the values of HPCN. It is of utmost importance for the Swiss research community to explore and unleash the tremendous potential HPCN is providing. However it is in my opinion wrong, to jump ’blindly’ onto the Petaflop-Wagon. While it may be prestigious to have one of the first Petaflop systems, Switzerland is best served with a staged version which includes all the necessary boundary conditions. This integral view of the complex problem is necessary to achieve sustained Petaflop performance, which is our ultimate goal! We urgently need innovation and not scaling up existing technologies and eventually not holding to existing structures in the Swiss HPCN landscape . With this in mind I am looking forward to a Swiss Petascale Computing System which will give us at least three orders of magnitude more computing power. With the best regards Andreas Adelmann PAUL SCHERRER INSTITUT, CH-5232 Villigen PSI, Schweiz July 4, 2007 SNSP-HPCN | Final Report | Appendix G Seite 3 page G-8 SNSP-HPCN Open Day 6.12.06 ETH Board July 4, 2007 > > > SNSP-HPCN | Final Report | Appendix G > > The existing network infrastructure between academic sites should be used in the context of HPCN. The Swiss academic HPC network should stay within the responsibility of SWITCH. This network has the ability to scale with future needs. CSCS does fulfill a very well appreciated role in providing a reasonable supply of HPC ressources within Switzerland. Nevertheless, due to the fact that all of the Universities as well as the Fachhochschulen have the ability to run their own HPC centers, those are to be considered as equal partners. Thus we think, a partnership must include all Universities, Fachhochschulen, PH's. We think that this integration can be best managed by SWITCH. page G-9 Aeschlimann ETH Board HPCN Initiative – Comments from Ron Appel, Swiss Institute of bioinformatics and Geneva University 1. Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the top-performing computing system that Switzerland should acquire in 2008-2011? As often, there are two aspects to consider when deploying a large-scale nationwide project. On one hand, developing a nation-wide High-Performance Computing and Networking infrastructure will allow many important projects to become realizable and thus will enable Swiss academic and industrial research and development to remain at the forefront of their respective fields. On the other hand, such an infrastructure will require large amounts of funds to be immobilized in hardware, software and human resources and thus the flexibility in setting up new innovative environments and projects that is absolutely necessary in today’s science might become difficult to maintain. This being stated, Switzerland should build the necessary infrastructure, funding scheme and competencies so that it can provide the required HPCN environment and resources when specific project require them. 2. What is the optimal role of CSCS within the overall strategy and the ideal balance between such a role and other actions? The CSCS should: 1) host and provide a basic level of HPC resources (hardware and middleware); 2) Build and maintain a state-of-the-art competence center in order to be in the position of rapidly expanding available resources when necessary, or developing new innovative technology; 3) analyze the changing requirements of the Swiss scientific community and advise them on the best solutions to follow; 4) assist or assume the responsibility of such developments; 5) help other Swiss academic HPCN centers to build the necessary competence and to provide the required resources. 3. What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy – that should be considered while planning the acquisition and deployment of new HPCN facilities? HPCN must be planned, funded and coordinated at the national level in order to take into account several issues: 1) CPU resources; those must be at all time sufficient to sustain the various scientific projects undertaken in Switzerland. 2) GRID technology must be mastered in order to take advantage of the resources (not only hardware) tat are inevitably scattered around the country. 3) In areas such as Life Science, a Data Grid must be deployed to allow for easy access and exchange of the very high volume of data. 4) Competencies in middleware must be expanded, in particular to achieve the required security level. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-10 ETH Board 4. What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? HPCN should be made accessible to a larger public. Basically, the national HPCN infrastructure should be part of the available resources for most if not all scientific projects in most fields. 5. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? CSCS should provide resources and, as already stated, more importantly competencies to the scientific community at large, by following guidelines defined be the community itself. This could be achieved by forming specific working groups in the various fields that will evaluate and anticipate the requirements for HPCN resources. Workshops could be organized to publicize the existence of such working groups. 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific financial credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc…)? Such procedures look reasonable, but other similar procedures in other countries should carefully be studied first. 7. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? Switzerland is a federative country. Similarly, HPCN would be most useful if it was to be built in a federative way: avoid centralization of resources, centralize competencies and develop a procedure to make these competencies available nationwide. 8. How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? Again, the scientific application groups should guide the development ans planning of any HPCN initiative.. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-11 ETH Board Input Kim Baldridge 061202 Dear Marie-Christine I unfortunately will not be able to attend the Open Day meeting in Bern on December 6, but am very interested in initiatives being discussed. Below I have replied to the questions that were posed in advance of this meeting, and made some additional comments. Thank you in advance for considering this input on behalf of Wibke Sudholt and myself. 0) I, as others have already expressed, think it quite important to include member representation from the Swiss Universities in this initiative, particularly if this is oriented towards a Swiss National Strategy. Given the many efforts and interested parties in the Universities, this should not be a problem. 1. Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the top-performing computing system that Switzerland should acquire in 2008-2011? To decide this question, a careful and detailed analysis of the current situation of computational processing in the Swiss HPC centers, in particular at CSCS, should be performed. Which of the submitted jobs require an "expensive" supercomputer infrastructure, which could/should be executed on a "cheaper" cluster or grid infrastructure? How much necessary processing time cannot be granted to projects that are scientifically interesting? What could/should better run on individual University resources? How can the future needs be approximated by using current and previous such data? While there are several specific scientific domains that can exploit very high end computational infrastructure of the 'tera' and 'peta' scale, many areas of computing, petaflop-level systems are not efficient or employable, yet, access to general-purpose and commodity-style hardware, and grid oriented infrastructure is highly useful. Given strategic planning, it would be possible to bring a few of these latter domains into the higher end, but not without resources/people to restructure algorithms/applications. 2. What is the optimal role of CSCS within the overall strategy and the ideal balance between such a role and other actions? CSCS should host and maintain any central HPC systems (supercomputers) in Switzerland that are accessible by all research institutions in Switzerland. Furthermore, it should provide training and consulting for its users. Research projects should be performed in close collaboration with academic or industry partners. Furthermore, CSCS should provide project management and coordination for Swiss and international collaborations in the field of HPCN. The Swiss Grid Initiative provides an example for this. To fulfill this role in a balanced and fair way, it is essential for CSCS to become a neutral institution independent from ETHZ. Also, the assignment of compute time should - as far as possible - already happen when proposals are evaluated at SNF etc. Of course, expertise from CSCS will be needed to rate the proposed HPCN requirements. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-12 ETH Board 3. What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy - that should be considered while planning the acquisition and deployment of new HPCN facilities? Forthis question, a detailed analysis of the current and future computing requirements according to the one described under point 1 needs to be performed. Also, as mentioned in (1), there are many technical hurdles to bringing many applications into a form able to exploit petascale computing, and other domains need even more effort to construct algorithms appropriate for their domain, or exploit high end data facilities. Issues on the petascale bring forth serious issues with regard to fault tolerance and error analysis, as well as now, a higher level of data transfer and storage requirements, all of which must be addressed and a strategy in hand before production. 4. What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? Because of the limited number of users really ready for the 'petascale' level, what is needed is a 'ramp up' plan to slowly enable promsising fields and scientific domains. This means, support for a hierarchy of high end compute facilities, together with support for software and middleware. Instead of solely supporting the installation of a big central hardware, the idea of having a reasonable set of of modest-to-high end hardware and in particular the development and maintenance of application software and computational and data middleware at research groups and Universities should be better supported. This especially means a better and longer-term support for people. The necessary sustainability in system administration and software development cannot only be achieved by PhD students and temporary project staff, as this means a constant loss of knowledge due to the high turnovers created by this situation. Instead, core persons should be employed in sustainable ways. Furthermore, the interdisciplinary collaboration between infrastructure providers (University IT service centers, CSCS, SWITCH etc.), domain research groups, and computer science departments has to be significantly intensified, both within each academic institution, as well as nationally. The Swiss Grid Initiative provides an example for this. Key HPCN scientific research domains and projects that can already or have the possibility of exploiting high end, or, are worth major effort due to scientific payback, should be invested in in terms of software development or/and adaptation. An end-to-end process should be set up that includes involved partners, for coupled scientific and computational domain expertise. The hope from this is the production of better software, a better training of the involved people, as well as a better usage of the available resources, for an outcome of interesting research developments. 5. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? Consulting, collaboration, and training have already been mentioned in point 2. One way to enable this would be financial support for mutual collaboration and visits of University researchers and HPCN consultants between the different institutions, to enable the exchange of knowledge. Furthermore, national working groups should be set up that include representatives from both HPCN and domain research, and that focus on July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-13 ETH Board different aspects of HPCN and scientific computing in general. Such working groups need to have some sort of budget to really enable progress, however. There are many working groups now, with interested partners putting effort in out of good will, but in reality, with day-jobs, this becomes more and more difficult, and prolongs progress that could be made. 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific financial credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etcŠ)? We are not sure if credit points are the right instrument. However, when funding and staffing research groups and when evaluating scientific proposals, the issues described in point 4 need to be taken into account. With the rising complexity and importance of computational science and HPCN, we cannot afford anymore to have hardware maintenance and software development being done by graduate students in their spare time, or worked into current work-loads on a time-permitted basis. Instead, a more professional treatment of these aspects is highly required. A key point here is the support for maintenance of software and infrastructure, not just development. Too many efforts are extended to build near-production level software and support tools, and then due to lack of funding die off, leaving it to the next group with funding to either pick up or, more often, reinvent. 7. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? The main product of Switzerland is intellect, not petaflop. Therefore, the focus should be more on the holding, gaining, and support of people and the excellence of science than on new supercomputers. Excellent people and knowledge exchange both in the academic as well as in the industrial sector will allow us to stay competitive for a much longer time than a single new supercomputer. Of course, high-end compute resources can also catalyze knowledge gain and scientific excellence, but regarding hardware this is of lower strategic importance for Switzerland than regarding software and domain science. Furthermore, it is also important to accelerate the cooperation of academia and industry, in particular, by allowing people to move easily from one to the other side, exchanging their HPCN knowledge between both sides. 8. How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? The potential of scientific computing can already be seen to be of high importance in most all fields of science. This is also more and more the case for experimental research. This means that user interfaces of scientific application codes, data and other resources, and the usage of HPCN facilities have to become considerably simpler and more transparent (e.g., via portals and advanced client software). Researchers then can fully concentrate on their domain science instead of needing to worry about HPCN issues and will ultimately be much more productive in there respective research fields, thereby increasing the competitiveness of Swiss science. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-14 ETH Board Input Bednarek Ad.3 Performance issues and problems: 1. Code optimization and proper benchmark tests. There were many cases when vector computers were slower than average PC. Libraries optimization, proper compilation – a lot of human work involved 2. Disk and I/O performance – many software packages needs huge I/O efficiency. Queuing system should be aware and redirect this jobs to nodes with such capacity – efficient local disk system (if not, only few percent of CPU will be used). 3. CPU time wasting: Many jobs can run forever due to convergence problem. Certain time limits and CPU credits should be used. Owners of longer jobs should be reminded by emails to check progress of their work. It should exist possibility of CPU time extension, since some jobs can’t be restarted and CPU time would be lost. 4. Parallelization issues 5. Jobs submitted to the queue should be executed immediately (they can be suspended later if necessary). Longer jobs should have lower priority over short and normal jobs. Ad. 5 Stronger collaboration and links should be developed between similar research groups, facilitating problem solving, in form of web base discussion forums or similar. Ad. 6 Project accounts, credits points and CPU accounting should be used to facilitate work and evenly distribute resources. Check point 3.3 Simplification of proposals. CSCS could provide expertise and trainings with code optimization and development. Grid computing projects should get stronger, since thousands of desktop computers are idle during the night and most of the day. I think that cluster solutions are more price efficient than big vector computers and this kind of system are more flexible. Of course mixed systems are also very useful. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-15 ETH Board Richard Bührer, Direktionspräsident FHNW Swiss National Strategic Plan for High Performance Computing and Networking, SNSP-HPCN Statements for the upcoming OPEN DAY on December 6, 2006 General remarks (view of the Universities of Applied Sciences, i.e. Fachhochschulen) The Universities of Applied Sciences (UAS) in Switzerland heavily focus on application orientation (teaching, applied research and development). In order to provide high quality products the UAS base heavily on results from basic research being done at Universities and ETH. It is thus of great importance that basic research in Switzerland is done at top class level. For various research topics, the availability of high performance computing and networking at University level is thus fundamental. With respect to Universities and ETH, however, the availability of a petaflop system for UAS projects will be of much less importance. Strategy High Performance Computing at petaflop level should be based on an appropriate national strategy. This strategy must describe the customer views and demands (both academic and non-academic users), possible architectures of high performance computing and networking systems, organizational and financial aspects, timetable etc. It is obvious, that such a strategy will not cover all high performance systems since each academic organization is basically free to choose its own equipment. Unless at least the key aspects of such a strategy are available and validated, it is rather crucial to focus solely on one specific solution such as the upgrading of the CSCS to petaflop level. Selected Statements • A national strategy must outline the need for a petaflop-level system versus a concept of e.g. several sub-petaflop systems located in or close to selected universities (distributed concept). • Depending on the strategy, especially in case of a distributed concept, the role of CSCS has to be adjusted or redefined. An adequate user support has to be guaranteed in any case. • In terms of nationwide networking, data broadcasting etc. the potential and the role of SWITCH has to be carefully taken into account. • The definition of the most effective measures to enhance the academic contributions to HPCN in Switzerland is part of the national strategy mentioned above. Brugg, 1.12.2006 Direktion July 4, 2007 SNSP-HPCN | Final Report | Appendix G Schulthess-Allee 1 5201 Brugg T +41 56 462 43 70 F +41 56 462 44 01 richard.buehrer@fhnw.ch www.fhnw.ch page G-16 ETH Board Input Ueli Bundi, Eawag Dear Ralph, Here our comment to your questions in your letter of 2 November. We focus our answers on points 1 and 2. With best regards, Ueli Bundi 1. Should Switzerland acquire a petaflop-level system..... ? We are convinced that petaflop-level systems will play an important role for many disciplines in the future, e.g. bio and material sciences, weather and climate, hydrodynamics etc. Also from the national point of view Switzerland should profit playing an active part in this leading edge business. 2. What is the optimal role of CSCS.... ? Petaflop systems coming up in the near future will be highly specialized, built for solving just a single type of problems in an optimal way. The way towards a general purpose petaflop system which runs a broad spectrum of scientific codes will be very long. However, those specialized systems must be developed and used in close connection with the centers of application. It is obvious that CSCS as national service provider for high performance but general purpose computing is NOT the optimal place for such systems. A national strategy should allocate petaflop resources at research centers and in research topics of high priority. On the other hand the "low end HPC" market segment of CSCS will erode because of the dramatic improvement of price/performance ratio of general purpose compute clusters built on cheap commodity hardware which can be easily run at many sites. This "low end teraflop segment" will clearly dominate the progress of scientific and industrial high performance computing in the near future. We think the optimal role of CSCS can only be to fill the gap between the low end teraflop and the petaflop segment, in other words to provide services for the high end teraflop market for a broad community of scientific users. However, it is diffcult to decide how efficient investments into this latter market segment are. It may rapidly shrink because of asynchronous development of the segments below and above. Therefore we recommend not only to look for the needs for petaflop computing but to include the whole range of HPC needs of the Swiss scientific community before making new investments (despite of any political "boundary condition"). °°° Ueli Bundi Eawag Überlandstrasse 133 P.O. Box 611 8600 Dübendorf Switzerland Phone: +41 (0)44 823 50 21 Fax: +41 (0)44 823 53 98 bundi@eawag.ch http://www.eawag.ch July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-17 ETH Board Input Chopard, Bastien Cher Ernest, Suite à mon coup de fil de ce matin, je te donne brièvement ci-dessous et en anglais, ma position sur le sujet du plan strategic HPCN. I also think it is important that Switzerland adopt a clear position in the field of HPCN. A development policy in accordance to the role Switzerland wants to occupy in the international community is needed. There are numerous scientific and industrial challenges But, beyond that, it is of great importance that Swiss to high-end technology and keep developing a knowhow on reputation of excellence. We need to be in the front of avoid being at the tail. that requires HPCN. scientists have access which we build our HPCN activities to The CSCS must play a central role in such a project. But all Universities, EPF, Technical school should be associated in a balance way. GRID technology must also be reinforced and supported by the Swiss government. Having a clear message from the Federal government is also instrumental to develop and coordinate existing projects at a local level and stimulate research institution to encourage local HPCN initiatives. I also think that financing a large computing infrastructure is important but new positions for young researchers and engineers must come together to exploit the potential, ensure novelty and continuity. Avec mes amicales salutations, Bastien ======================================================================== Prof. Bastien Chopard University of Geneva, CUI Tel: +41 (22) 379 76 23 Departement d'Informatique Fax: +41 (22) 379 77 80 24 rue General-Dufour http://cuiwww.unige.ch/~chopard/home.html CH-1211 Geneva 4, Switzerland ======================================================================= July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-18 ETH Board Memorandum Date: From: Phone: Local: e-mail: November 27, 2006 Stephan Egli +41-056-310-3626 stephan.egli@psi.ch To: Prof. Ralph A. Eichler ralph.eichler@psi.ch Cc: Dr. Kurt Baltensperger kurt.baltensperger@ethrat.ch Swiss National Strategic Plan for High Performance Computing and Networking S. Egli, PSI Today scientific computing is often seen as a “third leg” of science, parallel to experimental and theoretical sciences. Beyond that scientific computing will become pervasive to science in general, thereby enabling new kinds of sciences or augmenting the current scientific technologies. There is a need for computing power at different levels of sophistication and magnitude and there is as well a need for a network of people knowledgeable in HPCN technologies. Scientists should either acquire HPCN knowledge themselves or they should team up with HPCN experts in order to tackle the most demanding challenges in science. Consequently the HPCN landscape in Switzerland should consist of a "pyramid" of machines, with a few high end capability machines of different architectures at the "top" combined with distributed HPCN resources throughout Switzerland building different tiers of this pyramid. In order that swiss research stays competitive to international research the aggregated computing power in Switzerland in 2011 should amount to about one petaflop, under the assumptions that the current exponential growth oh the computing power continues during this time period. However a need for a single one petaflop machine in Switzerland until 2011 is not likely. The operation of such a machine would be very expensive and risky, because many problems in operating system and hardware design would first have to be overcome in order to make efficient use of such a machine. The need for such a machine could only arise from an outstanding scientific challenge, which would have to be defined first. The role of CSCS should foremost be a service provider of machines from the top of this pyramid. It should provide infrastructure services for the scientific computing community in Switzerland. In addition CSCS should lower the threshold for making optimal use of this infrastructure e.g. by organizing courses, by helping to migrate existing codes to the supported architectures, or by providing dedicated experts which can be "rented" by the science community. It should help to organize conferences in order to strengthen the July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-19 ETH Board -2- connection between the swiss centers of the scientific computing community and the international centers. CSCS should be the competence center for procurement of high end computers, continuously pursuing research on market trends and collecting vendor independent technical expertise and experience. End users should be strongly involved both in the requirement definition and acceptance phase of these procurements. CSCS should furthermore coordinate all major HPCN procurement plans in Switzerland, thus helping to make optimal use of available Swiss resources. The prime goal is to foster state of the art computing technology and algorithms for science and guaranteeing continuity in the evolution of the most demanding machines. The main technical challenges to be expected for future machines are the reduction of the overall power consumption and the reduction of the mean time between failures by providing both components of very high individual reliability and by building from the beginning failure resilience into the architecture of these new machines both at the level of the hardware, the operating systems and the algorithms and software libraries. This century will be marked by our ability to look at complex and interacting systems requiring expertise from many disciplines to solve. Therefore investment in education of people and interdisciplinary research is even more important than investing in hardware. In particular a dedicated PhD program for scientific computing should be created. The expertise should be distributed over Switzerland and cluster around the existing science centers like ETHs, Universities and Research Centers. Team building of HPCN experts and scientists should be encouraged. All these activities should be closely aligned with the European and global activities in the HPCN area. The swiss “pyramid” of machines should become part of an even larger international pyramid of machines and experts, for tasks which are beyond the capabilities of a single country. There should be an explicit strong coupling of the HPCN strategy to the European future research activities and the building of new research infrastructures, as outlined in the recent ESFRI Report 2006 (European Roadmap For Research Infrastructures). July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-20 ETH Board Basel, December 2, 2006 Dean Flanders, Head of Informatics Friedrich Miescher Institute for Biomedical Research a part of the Novartis Research Foundation Maulbeerstrasse 66 Basel, CH-4058 Subject: Response to request for feedback on the Swiss National Strategic Plan for High Performance Computing and Networking (SNSP-HPCN) Dear Manuel, I have yet to see a supercomputer center or grid succeed in the democratization of high performance computing (HPC) and we should learn from these failures. CSCS needs to take a lead role in HPC in Switzerland, and facilitate HPC and its use (but not control it). However, as we already know, if a large single computer system is built, it will only be used to answer a limited number of big questions, inaccessible to most researchers, completely saturated (no matter how many CPUs), and be outdated the minute it enters the computing center. This HPC strategy should be different, it should be pragmatic, involve scientists in the process, focus on solving scientific problems (not developing new technologies), have no “sacred cows” (be open to commercial vendors of software), and not re-invent wheel. That is why I recommend a nodal approach, bringing compute resources near to researchers (and their data) and providing a broad portfolio of compute resources. The goal should not be a petaflop computer, but a petaflop of computing. I feel at the moment there are very few technical issues to overcome for this to happen, it is more of an implementation (and political) issue to establish a robust HPC infrastructure in Switzerland Steps for success: - CSCS should function as the location for highly specialized compute resources, however, this should be developed in a cost sharing scheme with research groups and no system should be implemented without a demonstrated need. - CSCS should serve as a repository for applications, center of excellence for HPC hardware (hardware group, 2 FTEs), HPC software standards (software group, 2 FTEs), and coordinate HPC projects (coordinator, 1 FTE). - CSCS should not try to become an expert in each domain of HPC use, but instead focus on empowering HPC users. - Creation of a user community (ideally a foundation) of HPC users (Swiss Grid). - Institutions can join this community, and subsequently groups of people from these institutions can join this community as a node. However, the node must contribute resources, follow certain standards, and have a connection to the SWITCH backbone. - A general assembly (GA) can be formed from the members of these nodes, and an executive board (EB) selected to represent their interests. - Special working groups to focus on research specific applications, e.g. chemistry, bioinformatics, systems biology, etc. should be formed from this community. These working groups should have a lead person (typically external from CSCS) and be responsible for application clearing (along with the hardware group and software group from CSCS) - Communication to the EB should come primarily from working groups (including the hardware group and software group at CSCS). - Yearly blueprints should be made (3 year time horizon) by this community. - SNF grant funding for computing can be tied to nodes participating in this community, and funded research groups can chose which node to associate with. - Data collected with SNF funds can be stored in repositories at their respective node. - Mechanisms must also be put into place to prevent saturation of compute resources. With this approach the heavy users get a vast amount of computing power at their disposal, and at a certain point, we will reach the critical mass to answer many questions nearly instantly and ask questions that would otherwise not be possible to answer. Kind regards, Dean July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-21 ETH Board Input Doris Folini Dear Dr Margartondo Dear Dr Baltensperger I plan to attend the open day meeting on December 6, 2006. Written statement, addressing key questions 4, 6, and 8: Well trained people are decisive to fully exploit the possibilities offered by HPCN. Key question 6 (specific credit points in the area of HPCN) thus should be answered by a clear yes. This also is part of the answer to key question 4: an effective measure to enhance academic contributions to HPCN is the education and subsequent scientific promotion of people that are trained to develop and use HPCN in order to answer scientific questions arising in their field of specialization. To key question 8 (HPCN enabling national services): Empa currently has a large user project (LUP) running at CSCS, which forms part of a larger, combined (measurements and modeling) effort that is focused on the air quality in Switzerland. The project is supported by the Federal Office for the Environment. Yours sincerely, Doris Folini Dr. Doris Folini Laboratory for Air Pollution / Environmental Technology Empa - Materials Science & Technology Ueberlandstrasse 129 CH-8600 Duebendorf Switzerland Tel: +41 (0) 1 823 49 74 Fax: +41 (0) 1 821 62 44 doris.folini@empa.ch www.empa.ch July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-22 ETH Board Prof. Antoine Geissbuhler Directeur du département de radiologie et informatique médicale Ligne directe: 022 372 62 01 antoine.geissbuhler@hcuge.ch High Performance Computing and Networking (HPCN) needs, a medical informatics perspective The convergence of biomedical and information sciences is bringing new challenges, in terms of knowledge representation, ability to establish new connections, and requirement to support new computing needs. The two latter challenges are related to HPCN. First: connectivity. The ability to connect medical institutions is becoming a key factor in the success of federating information, in order to reach the critical mass necessary for advanced th fundamental and clinical research. Such networks are being promoted in the EU 6 (e.g. the th AneurIST project) and the upcoming 7 framework programmes, and are now mostly evolving towards grid-based architectures. It is essential that Swiss institutions are enabled to participate to these efforts. Usually, connectivity is more of an issue than bandwidth, even though real-time applications dealing with the identification of similar cases across the network, or with distributed data-mining applications, will require substantial networking performance, in terms of bandwidth and data protection functionalities. Second, computing power. Automated knowledge-acquisition tools and simulation tools are known to be computing-resource intensive. In a biomedical world that connects multiple knowledge domains (genomics, proteomics, physiomics, clinical sciences, population-wide information) and aims at being able to apply knowledge-discovery application techniques across them, it becomes important to have access to high-performance computing. Moreover, the following applications will require high-level computing power: - the ability to simulate patient-specific implants and therapies by combining clinical information, medical imaging, and flow simulation techniques, for patient-adapted therapies and for care professionals training, - the optimization of complex care production processes, using the continuous measurement of clinical activities, - the application of data-mining techniques in very large, federated clinical warehouses containing multimedia information. In some situations, it is foreseeable that these capabilities will be needed in real-time, to enable useful clinical decision-making and care processes. The Swiss strategy for developing HPCN should take these needs into account, and, in particular, make sure that translational activities, which bridge the gap between fundamental research, clinical research and actual patient care, are enabled. This is essential as a technology developed for basic research needs to be translated into patient-level and/or population-level benefits. Antoine Geissbuhler, MD July 4, 2007 HUG – 24, rue Micheli-du-Crest - CH-1211 Genève 14 Tél. 022 372 62 01 - Fax 022 372 62 55 - www.unige.ch/medecine SNSP-HPCN | Final Report | Appendix G page G-23 ETH Board Universität Zürich Informatikdienste 1. Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the top- performing computing system that Switzerland should acquire in 2008-2011? If the question would have been: does Switzerland need compute capacity in the range of Petaflops, my answer would clearly be yes. Regarding one monolithic petaflop-system, I have certain doubts. Is there a demand for such a system? How many user groups could efficiently use such a system within the given time frame? Is there or will there be software, OpenSource or commercial that can utilize such a system? Is there know-how to develop new and port and adapt existing software to such a system? Are there enough human resources and enough experts in the field of HPCN to prepare for such a system within the given time frame? Regarding the enormous costs of one petaflop-system project, the available funds could also be invested in several smaller sub-petaflop-systems. In my opinion, a grid like approach like Swiss Grid with several distributed systems is more promising. These distributed systems could be a) optimized to fit special demands of user groups regarding number of nodes, interconnect speed, memory and fast access to hundreds of Terabytes of storage and b) with such different systems, Swiss researchers and students could learn how to utilize huge compute capacities in their research field. If this ecosystem of experts, software, hardware, teaching and communication exists, if the Swiss HPCN community is ready, the acquisition of a single petaflop-system could be reevaluated. Another consideration regarding petaflop-systems is, that after maybe four to six years, the infrastructure costs (energy, cooling, floor space etc.) for such a system will become prohibitive compared to then available new and faster systems. Investments in a petaflop-system are not one time investments on a project basis. If Switzerland engages in a petaflopsystem continued financing for this infrastructure has to be guaranteed. 2. What is the optimal role of CSCS within the overall strategy and the ideal balance between such a role and other actions? CSCS has the expertise to run big machines. It should provide access to high-end compute resources that academic institutions can not afford to buy and/or run. CSCS should acquire the expertise needed to empower users at research institutions and industry to fully utilize high-end compute resources (software development, system administration etc.). CSCS could also act as a coordinator to establish a Switzerland wide collaboration and know-how transfer between research groups and industry. 3. What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy – that should be considered while planning the acquisition and deployment of new HPCN facilities? The term “petaflop-system” might be misleading as petaflops are just one of many – and in my opinion not even the most relevant - features describing a high-end computing system. For applications to run fast, different aspects that influence the speed have to be taken into account. Most important besides CPU interconnects and amount of memory is IObandwidth/performance. For many applications that deal with huge amounts of data (e.g. Systems Biology applications, data mining applications), the access to the data is the limiting factor, not the numbers of Flops a number of CPUs can do. A petaflop-system has to take these requirements into account. So, investments not only have to be made in compute nodes but also in a very high-end petabyte storage infrastructure. As data should be kept close to the analysis system and because one central system is more economical than several distributed systems, an infrastructure for data safekeeping including data and location redundancy has to be included in the petaflop-project. To keep the data archives at a reasonable size, the very strict management of these stored data is of July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-24 ETH Board Universität Zürich Informatikdienste utmost importance. An audited infrastructure for the encryption of transmitted and stored sensitive data has to be provided, especially for industry. To deliver data from different sites within a reasonable time frame, network connections from data producing systems to the central data analyzing petaflop-system must be adequate. Switch can provide this infrastructure. 4. What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? Teaching programs at different Swiss educational institutions have to be started as soon as possible. A very important factor for the sustainability of Swiss HPCN efforts are people. A nation wide strategy can not be based on only the work of PhD students or Postdocs. The fluctuations of these scientists are always accompanied by loss of know-how. Long-term jobs have to be generated for software developers and system administrators, preferably in collaboration with local IT-services that already have a long history of providing services to science. Optimization and maintenance of middleware and software are essential for HPCN. Unfortunately, such work can not be published and funds for this kind of work are very hard to find. A Switzerland wide HPCN strategy has to take this into account and funding for this work has to be provided. 5. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? The IT-departments of the different HPCN centers and CSCS have to cooperate closely in the general plan for a Swiss wide HPCN collaboration project. Often, science looses interest in a compute project as soon as results can be produced. Further development of a system to become usable to more than one group are not interesting and funds are missing to turn a project into a service. Unpleasant considerations regarding Authentication, Authorization, Accounting, Availability and Security are more in the field of the IT-departments that of science but they nevertheless have to be taken into account. 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific financial credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc.)? In my opinion this is absolutely necessary to form the basis of a long term HPCN strategy. 7. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? Presently, a rather small group of highly trained scientists is using HPCN in Switzerland. In the future, new types of users will emerge that need access to HPCN. These users will come from fields not yet using or just starting using HPCN. Those user groups will emerge from institutions like philosophical faculties, banking institutes, medicine and so forth. Those users are not computer experts so they will need very easy and some times real time (e.g. medical imaging) access to HPCN resources. How fast and easily these users can access HPCN resources depends on the development of easy to use but powerful portals to HPCN resources. Mistakes that must be avoided In my opinion, a “one solution fits everything” strategy is a very expensive strategy and prestige should not be a driving factor for HPCN in Switzerland. Demands and number of July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-25 ETH Board Universität Zürich Informatikdienste users should be carefully evaluated. A compute Grid like solution (Pyramid) like proposed by the Swiss Grid Initiative/Association should also be evaluated. A HPCN strategy can not only be science driven. The different administrative bodies of Universities, Fachhochschulen and other interested parties have to be included to guarantee the sustainability of the strategy by providing the necessary infrastructure and human resources. Funds should be available also for these efforts. One question is: Should the HPCN project be regarded as a science project or as a project enabling science. In my opinion, although a certain part of the project must be focused on scientific research of the computing infrastructure and new methods using it, the main focus should be on enabling science. A group of experts should provide the interface between science and infrastructure. 8. How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? With a strong HPCN community in Switzerland scientific applications can be much more easily developed if researchers do not have to worry about underlying HPCN issues like Distribution, Resources, Authentication, Authorization, Accounting, Availability and Security. Productivity will rise, new services can be provided on national and international level. Switzerland (Swiss Academic institutions) will be able to attract more high-profile scientists from different fields thereby becoming more competitive in the international science and education “market”. Which again draws more funding and toplevel science to Switzerland. 12.12.06 / A. Godknecht / IT Services Universität Zürich July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-26 ETH Board Answers to the Questionnaire on SNSP - HPCN =========================================== Christoph Grab, Institute of Particle Physics, ETH Zurich. Preamble: The "classical thinking" of computing hierarchies envisions the concept of a a pyramid of computing system: - at the top is a single very high-performance machine (e.g. 1-10 PFlop type), * serving just a few users at a very high cost/user; * single geographic locality: has the risk of damage (earthquake, fire...); gain in terms of networking, data movement; * efficient exploitation needs concentration of knowledge for operation, for software implementation and application groups. * difficult to gradually evolve with technology. - the bottom may be federation of many distributed, smaller, cheap, commodity compute clusters, serving a large user base. * being distributed, systems gain in terms reliability (easily replacable), redundancy; * drawback is networking, data movement; * knowledge is spread, but communication between expert is more difficult; but knowledge is closer to the users, and can profit from the local know-how of users; * such systems can be very easily evolved and grown by the community itself (e.g. additions are possible locally at any University/ Institute/ Industry). - in between these two extremes, there are many possibilities for dedicated systems; such as 10 TFlop machines (e.g. the Horzion cluster at CSCS). However, new ways of envisioning this pyramidal structure could involve an overall "GRID-like" view, where the two extremes are both part of an encompassing new computing model, which goes beyond the single HPCN or the "bottom-level large-scale clusters". Therefore, it could be a unique opportunity for having a general Swiss strategy for HPCN to unify these two approaches into one single "Swiss computing and networking environment", where all these approaches coexist and benefit from each other. ----------------------------------------------------------------------------With this preamble, here are my answers: ======================================== 1)Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the top-performing computing system that Switzerland should acquire in 2008-2011? A:ONE single PFlop machine serves always only a limited number of users, who must have a very strong argument that they need it. Therefore an acquistion should be driven by a strong scientific case. I do not believe, that in general, buying the infrastructure will spur the scientific case, but it should happen the other way around. As soon as there is a strong scientific case, I immediately would support the acquisition of such a machine. We, the particle physics community, would not really profit from such a PFlop machine. It is probably more advisable to have a few mid-level systems (of order 10-100 TFlops each), that are still high-end, but can serve a wider group of users. ----------------------------------------------------------------------------2) What is the optimal role of CSCS within the overall strategy and the ideal balance between such a role and other actions? July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-27 ETH Board A: CSCS could operate the top Swiss-machines (be it a few mid-level, or a single top-level system). Through this, they could provide the expertise and competence to all the user communities. CSCS could provide a "pool of experts" to be involved in particular scientific projects; these experts could be integrated into the scientific teams, located at the institutes/Universities for certain time-periods, depending on the project needs. (E.g. one could think of a model like the "consulting companies", where the "teams" are put together out of different pools of persons and expertise). ----------------------------------------------------------------------------3) What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy - that should be considered while planning the acquisition and deployment of new HPCN facilities? A: For any applications, redundancy and reliability are the most important issues. - data storage, ie. "self-healing" systems; expert systems for supervision. - for data transfer, ie. multiple networks operating independently and providing redundancy. - having multiple systems at different geographical locations helps for the case of physical protection. ----------------------------------------------------------------------------4) What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? A: Most important: involve young, clever and motivated students as much as possible. This could be achieved through: - setting up dedicated PhD programs; setting up long term collaborations with research groups at the ETH/Uni. - being geographically and logically linked to academic institutions (ETHs, Unis), such that the every day communication is fostered. - advertising "scientific challenges" to be solved by "modern computing means" with apropriate rewards. - give (some promising, but not all) students also access to the high-end machines for "creativity". ----------------------------------------------------------------------------5) How should CSCS and the other HPCN centers be linked with the scientific community and leadership? A: One could set up joint groups for various purposes: - every day operation issues, - specific research issues ... ----------------------------------------------------------------------------6) Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific financial credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc...)? A: I am not completely sure, I understand the question: is a "financial credit point" to be understood as something like a predefined allocation of a certain amount of "CPU-time per months per research group"? Yes there will be a need for a resource allocation policy. but, this should be needed mostly for a single TOP-level machine, rather than for the base-level federation of clusters. ----------------------------------------------------------------------------7) How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? A: The industrial competitiveness could be enhanced by selling resources (e.g. allocation time on the top-machines) to the industry. They could even contribute themselves by adding machines, sponsor development/research projects, or getting actively involved in research projects. A: national security could be impacted in the fields of weather forecasts, glacier movements, earthquake safety, spreading of diseases etc. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-28 ETH Board by results of particular research projects exploiting the top-scale machines. What are the key mistakes that must be avoided? A: One of the key problem today is the slow, and highly burocratic way of acquiring computing resources. a) funding e.g. from NSF is only YEARLY granted, which makes it extremely difficult for long term planning of large computing systems. b) the present RFP (request for proposal, oeffentliche Ausschreibung) procedure is not suiting the needs for acquiring large-scale computing resources at all. ----------------------------------------------------------------------------8) How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? A: By easing the technical aspects of their computational challenges, and therefore allowing them to concentrate on the scientific aspects, ie. take the load of technical details off the shoulders of the scientists. This could be in the fields of generic interface services, authorisation, authentification services etc. They may produce eventually spin-off systems that can be exploited for national services, such as e-banking, e-voting, credit-card system, transport access systems etc. ----------------------------------------------------------------------------- July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-29 ETH Board Swiss National Strategic Plan for High Performance Computing and Networking, SNSP-HPCN Questionnaire 1. Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the top-performing computing system that Switzerland should acquire in 2008-2011? Yes, Switzerland should acquire a petaflop-level system. I think one scientific justification in the life science sector is the triggered ‘paradigm shift’ in drug discovery. With a petaflop-level system multi-dimensional in silico compound screening experiments might become possible simulating the interactions between a disease target and all known chemical compounds to predict which compounds participate in desired interactions. These simulations and predictions are currently not possible. In an ideal world a graduate student sends the chemical structure of a disease target (along with some chemicals to be tested) to the petaflop-system which returns simulation results some minutes later. What is the optimal role of CSCS within the overall strategy and the ideal balance between such a role and other actions? CSCS could, besides running the machines, be responsible for setting up and hosting a software repository for embedded interoperable systems. A software engineering team on the order of 10 people would be required to get and keep collaborations going, to build up and maintain a repository at the high quality level required and to be perceived as ‘valuable’. Furthermore this team would need to work very closely together with the information and communication technology researchers in order to constantly adapt the software engineering processes and management methodologies to the needs of the different communities. What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy – that should be considered while planning the acquisition and deployment of new HPCN facilities? In hand with the petaflop-systems goes a cutting-edge networking infrastructure, potentially going to Tbit Swiss wide, post state-of-the-art (quantum-) cryptograpghy, redundant data handling is not a problem per se (Legitimate reasons for redundancy are for instance performance, robustness, back up), data analysis packages need to be migrated onto the repository and refactored and document to reach a mature level such that they become interoperable, the user community needs to be trained to write packages and how to use them. What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? A clear vision and mission, what Switzerland wants to accomplish until e.g. 2030 with the HPCN. Everybody needs to feel realigned to the vision, needs to be excited about contributing to the success of the initiative. People need to think about exciting new projects and challenges even in the spare time. So, that means an open transparent dialogue is required to form a community, to have the community develop a vision and do it. The community needs to be given the responsibility and the empowerment to produce a vision and an execution plan, get them signed off by the ETH Rat and then get the empowerment to go ahead. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? Through common projects, inferred from the HPCN vision. Leadership doesn’t mean charisma. So you don’t want to have charismatic people only, but real leaders, modest people who treat their peers with respect, who have excellent communication and conflict management skills, social facilitators, not ‘antagonizers’, with an excellent scientific background, not necessarily professors, people who live the vision of the greater good of the Swiss society. These people shall lead the initiative being responsible for the processes, but not necessarily for the content. The content must come from the researchers themselves. As this is a top-down approach, many researchers will feel “not being asked”, these people need to be heard and understood, too. A leader will know that he needs to listen to them. 2. 3. 4. 5. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-30 ETH Board 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific financial credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc…)? If the credit point system is the same for everyone, simple, clear and transparent, I think it’s worth being considered. But there must only be one system which is objective criteria. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? Today's information society is confronted by information overload and communication collapse, ever increasing complexity and dependency on information access, and information safeguards. Information has become a valuable resource, as such effective and secure management of information is key to maintaining our global leadership position in industry and research and securing our political stability. Therefore it behoves us as a Swiss society to make the corresponding investment in the research and development of information technologies to stay at the forefront of innovation. It must be avoided that there are several initiatives where one will antagonize the other. How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? I think it could have a huge impact on R&D in nano physics, mechanical and electronic engineering including information and communication technologies, but also life sciences, with the ultimate goal to successfully perform in silico ab-initio modeling of complex entities such as whole organisms. See also 7. th 7. 8. Adrian Honegger, November 11 2006 July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-31 ETH Board Stellungnahme Beat Hotz Hart Sehr geehrter Herr Baltensperger In einem Schreiben von Herrn Scaroni vom 3. November 2006 sind wir über die laufenden Arbeiten an "Swiss National Strategic Plan for HP Computing and Networking" informiert worden. Als designierter Leiter der KTI a.i. (ab kommendem 8. Dezember) nehme ich gerne dazu Stellung. Die Förderungsagentur für Innovation - KTI begrüsst die Entwicklung einer nationalen Strategie für das Hochleistungsrechnen in der Schweiz. Neben Theorie und Experiment übernehmen Modellierung und Simulation eine immer wichtigere Rolle in der Spitzenforschung. Im Rahmen der verfügbaren Mittel sowie in Einklang mit den laufenden Entwicklungen in Europa (z.B. anstehende Entscheide über Aufbau von Zentren des HPC) sollte die Schweiz deshalb geeignete Ressourcen für das Hochleistungsrechnen sowie entsprechende wissenschaftliche Beratung der Forschung zur Verfügung stellen. Die Ressourcen sollten aus Effizienzgründen wenigen, wissenschaftlich hervorragenden Projekten über einen Wettbewerbsmechanismus zugeteilt werden. Deren Resultate tragen einen wichtigen Beitrag zum wissenschaftlichen Output der Schweizer Forschung bei. Das Hochleistungsrechnen muss in der Schweiz unbedingt zwischen den relevanten Partnern CSCS, den Hochschulen und den Forschungsinstitutionen abgestimmt werden. Dabei sollte der Fokus auf die koordinierte Beschaffung und Benutzung der vernetzten Hochleistungsrechner (Supercomputers und Hochleistungsclusters) gelegt werden. Dass dabei Switch, an dem das BBT ebenfalls beteiligt ist, funktional sinnvoll in eine kollaborative Lösung integriert werden muss, versteht sich von selbst. Ich danke Ihnen für die Berücksichtigung dieser Stellungnahme. Mit freundlichen Grüssen Beat Hotz-Hart BUNDESAMT FÜR BERUFSBILDUNG UND TECHNOLOGIE BBT OFFICE FÉDÉRAL DE LA FORMATION PROFESSIONNELLE ET DE LA TECHNOLOGIE OFFT UFFICIO FEDERALE DELLA FORMAZIONE PROFESSIONALE E DELLA TECNOLOGIA UFFT Beat Hotz-Hart, Dr. Vizedirektor Professor der Universität Zürich Effingerstrasse 27, 3003 Bern Telefon ++41 (0)31 322 21 30 (Mobile ++41 (0)31 79 619 12 86 Fax ++41 (0)31 324 96 15 beat.hotz@bbt.admin.ch www.bbt.admin.ch July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-32 ETH Board Input Jörg Hutter Subject: RE: Input in Advance of the OPEN DAY on December 6 Dear Mrs Sawley let me start expressing my concern that a project group working on a Swiss National strategy for HPCN without a member representing the Swiss universities will come to a solution tailored for the ETHs only. Because of teaching duties I will not be able to attend the Open day meeting in Bern. Questionaire: 1) Should Switzerland acquire a petaflop-level system? I don't think the investments into hardware of about 350 MSfr (estimated from money allocated for IBM and Cray to build a petaflop computer in the US) necessary for such a system can be justified today. The lack of a user community ready to use such a machine, together with the (most likely) imbalance of money spent for hardware instead of software and human resources will lead to a failure and will harm HPCN on the long term. 2) What is the optimal role of CSCS? CSCS should host, maintain and run the largest HPCN resources in Switzerland. CSCS should coordinate and provide services to HPCN efforts located at Universities. CSCS should collaborate closely with other HPCN centers at Swiss Universities. 3) What are the technical issues? We can assume that a single thread or CPU will be able now and in the future to perform in the 10 GFlop range. This means a petaflop computer will have 100'000 CPUs. Assuming that a standard job will use 5-10% of the available resources, applications that can run 5'000-10'000 way parallel are needed. Only very few applications are ready to do so. In fact most applications need a change in paradigm for there basic algorithms to be able to reach this goal. Additional problems with data volumes produced on such systems and their storage and retrieval for analysis have to be solved. 4) What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? Software has a longer life cycle than Hardware. Therefore investments in software can have a much higher return value. In addition, investment in software could be done within Switzerland (not so for hardware), contributing to producing HPCN experts and having an impact on Swiss society. 5) How should CSCS and other HPCN centers be linked with the scientific community and leadership? Optimally, the scientific community should identify its HPCN needs and CSCS and other centers should provide the tools. Too often July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-33 ETH Board we see the reverse situation happen. 6) Should the ETH domain, Swiss universities establish specific financial credit points in the area of HPCN? World class computational simulations need adequate hardware and the best software. This type of science is supported by SNF and other funding agencies. Specific grants in this area will certainly be welcomed. On the other hand, software development and especially software maintenance isn't funded and is done by most research groups by "miss-using" application research grants. What is missing is the understanding that simulation software for HPCN is also infrastructure and should be supported in a similar fashion as hardware, meaning long-term commitments of funds. This is in contrast to short-term (2-4 years) funding of application projects. I also want to stress the point that this software infrastructure is provided by the computational scientists (in physics, chemistry, engineering) not computer science. 7) How will the HPCN strategy impact the industrial competitiveness of Switzerland? Projects in computational science will allow to keep Switzerland at the forefront of many scientific disciplines. Industry can directly benefit form the research and will have access to the researchers trained at the highest level. 8) How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? Switzerland has leading groups in the development of scientific software. Many other groups use this software or benefit from the methods developed in other ways. The research performed using HPCN impacts many other fields, especially experimental research is more and more dependent on simulations for the interpretation of data. This is true for the small scale, e.g. an organic chemist in her laboratory, as well as the large scale, e.g. high energy physics or astrophysics. with best regards Juerg Hutter ---------------------------------------------------------Juerg Hutter Phone : ++41 44 635 4491 Physical Chemistry Institute FAX : ++41 44 635 6838 University of Zurich E-mail: hutter@pci.unizh.ch Winterthurerstrasse 190 CH-8057 Zurich, Switzerland ---------------------------------------------------------- July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-34 ETH Board From: Tel.: E-mail Data Prof. Dr. Mehdi Jazayeri +41-(058)-666-4656 mehdi.jazayeri@unisi.ch 29.11.2006 Statement on SNSP-HPCN This is my statement responding to the request for comments in the letter dated 09.10.2006. My comments apply only to the points 2 through 6 in which I have some expertise or interest. 2) Role of CSCS: It is clear to me that CSCS should move from being a pure service organization that provides cycles to customers to a more scientific organization which provides expertise in software development to its users. Its know-how should be increased from just configuring/maintaining hardware to higher-level software considerations of how best to use the computing hardware to provide high performance to applications. This kind of expertise is very specialized and rare. At the same time that CSCS would be providing such valuable service, it would also create a scientific environment for its employees to grow. This kind of knowledge could enable scientists to get increased performance out of a hardware before new supercomputers are available. Each new supercomputer probably requires new software approaches. 3) Technical issues: In addition to the “hardware” issues, the increasing importance of software should be taken into account. That is, the availability of software methodologies, tools, and environments. 4) The academic contribution: The emphasis should be on software: methodologies, tools, environments, performance measurements, and so on. While the scientists are only interested in the results of their software experiments, CSCS should concentrate on the software scaffolding to help them get their results faster. 5) Linkage to scientific centers: CSCS should be linked closely to the local universities’ informatics departments so that its researchers can be involved in the academic community on a day-to-day basis. This would contribute to making CSCS an integral part of the scientific community rather than being only a service organization. 6) Academic credit units: Certainly if CSCS finds a way to define the elements of the software expertise that I have outlined above in a methodical way, and it is linked with a local university’s informatics department, a next logical step is to create a specialization master degree on the topics. Courses could be developed that would form electives in a general master’s degree or their own specialization. This, of course, would require changes and new programs at the associated university where such programs probably do not currently exist. But creating such programs in software for high-performance computing would certainly help the long-term development of high-performance computing in Switzerland. Mehdi Jazayeri Dean, Faculty of Informatics July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-35 ETH Board Comments from C. Victor Jongeneel, Swiss Institute of Bioinformatics 1. Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the top-performing computing system that Switzerland should acquire in 2008-2011? The acquisition of computer hardware should always be driven by specific projects, and not the other way around. If a project emerges that is of clear national interest and requires petafloplevel compute resources, then such an acquisition makes sense. If not, it doesn’t. In my specialty area (life science informatics) there are to my knowledge no projects at this time that would require general-purpose petaflop compute power. However, in the foreseeable future, life science research will move in the direction of computational simulation of ever-larger and more complex systems, and within 5-10 years will require this level of computing power, if not more. 2. What is the optimal role of CSCS within the overall strategy and the ideal balance between such a role and other actions? The roles of CSCS should be (1) to provide hardware and project support for scientific projects that require compute resources beyond what is locally available in their Universities or Institutes; (2) to host enough compute power to support the largest-scale projects envisioned within a 5year perspective; (3) to develop further high-level expertise in fields where it is already strong, and serve as a focal point for dissemination of this expertise within the academic and industrial fabric of Switzerland; (4) to act as an early adopter of new technologies to field-test them and provide sound technical advice to other IT centers in the country (a role which it has not taken so far); (5) to act as a leader in the development and implementation of Grid infrastructures in Switzerland. 3. What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy – that should be considered while planning the acquisition and deployment of new HPCN facilities? The main issue in my opinion is that the planning of HPCN facilities be coordinated at the national level, and that individual facilities funded by the Federal Government be integrated into a nationwide Grid. Switzerland already has an excellent networking infrastructure through SWITCH, and its bandwidth should be upgraded in response to observed and predicted network traffic. SWITCH should also provide a simple nationwide user authentication and security certificate infrastructure. While this is under way, the current mechanisms are too complex. Finally, issues of data redundancy and protection are so project-dependent that they can only be planned for at the project level. 4. What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? Swiss academic institutions, and especially the universities of applied sciences, should teach the specific skills that are necessary to program and maintain HPCN infrastructures. This includes system management skills for complex clustered environments, programming massively multithreaded systems, parallelization and scaling of existing code, etc. 5. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-36 ETH Board CSCS and the other centers should have a strong presence in all the Swiss academic institutions, by organizing seminars and presentations of their activities, and explaining to the community how HPCN can contribute in very concrete ways to scientific research. It would also be helpful if existing academic institutions did not compete with each other in establishing HPCN facilities, and thereby insulate themselves from other HPCN users. Now that there is a University in the Italian-speaking part of the country, CSCS should also strengthen its links with this University, e.g. by being formally linked to one of its Faculties and by taking a role in teaching computer science, at least at the graduate level. Stronger links could also be created by encouraging the establishment of more groups in the Ticino that use HPCN in their work. Finally, CSCS should take a leading role in developing the Grid infrastructure required to effectively link the HPCN centers to each other. 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific financial credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc…)? I assume that the question asks whether paid credits (“funny money” credits for computer time in my graduate student days) should be established for access to HPCN infrastructure by academic institutions. I think that this is a good idea in principle, because it makes institutions financially more responsible for the resources they use, and can help fund the HPCN infrastructures in the first place. However, such a system should be established only once the infrastructures are a proven success, and competition for access becomes an issue. 7. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? Access to HPCN facilities is essential for industrial competitiveness, especially for SMEs that cannot afford installations in house. In the commercial area, it is essential that security and confidentiality be assured at the infrastructure level (see question 3), both within individual HPCN facilities and through the Grids that connect them. The key mistakes to be avoided are: • • • • The concentration of all resources in a single center, even if one is pre-eminent The fragmentation of resources between centers that each consider themselves as the “owners” of the resource, rather than interdependent members of a network The acquisition of HPCN resources for prestige reasons only (e.g. appearing in the Top500), without having identified specific needs The tendency for Switzerland to “go it alone” in HPCN, and the under-use of large-scale resources available at the European level (fragmentation of the ERA, in EC-speak) 8. How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? Again, I am not sure what the question is. Scientific application groups should always be closely linked to HPCN facilities, and provide the interface between the facilities and the community at large. As much as possible, individual facilities and the application groups that work with them should have complementary domains of expertise, and not compete with each other. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-37 Supercomputing SC Strategic Plan SP (G. Lake) ETH Board NEED A STRATEGIC PLAN, an acquisition strategy is later Key ?: Monolith or distribute geographically and/or over machines tailored to project needs as many have suggested Large projects need to spend money early (typically 5-10%) to be effective later. Petaflop computers will not be “faster versions of what’s available now”. Need to invest in experimental architectures now and software to scale to 10100K parallel (maybe 10K nodes with 10-100 cores per node). If such experimentation is outside the scope of CSCS, there needs to be a special grant program July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-38 ETH Board Statement of Institute for Atmospheric and Climate Science ETH Zürich for the OPEN DAY MEETING, December 6, 2006 Our institute has been heavily engaged in Global and Regional Climate modeling for more than 15 years. To be competitive in this area we need access to state of the art HPC facilities. The following developments in our research field make a petaflop scale machine a desirable tool within the next few years 1. Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the top-performing computing system that Switzerland should acquire in 2008-2011? Our institute has been heavily involved in global and regional climate and atmospheric modeling for more than 15 years. We see the deployment of petaflop systems not as a quantum jump, but as a natural continuation of the past Swiss HPC strategy. To be more specific, the nominal compute power (Linpack) of current peak HPC installations in Switzerland amounts to between 7 and 18 TeraFlops. This is a factor of around 100 below petaflop performance, corresponding to between 1 to 2 hardware generations. There are exciting scientific prospects regarding the use of the next generation of atmospheric/climate models. The quest is beyond refining the spatial detail of current models, but about better representing the underlying physical and dynamical processes at the appropriate spatial scales. To make an example, most current atmospheric models rely on drastic approximations in their representation of convective clouds (e.g. thunderstorms). Spatial resolutions of O(1 km) are needed to account for the underlying spatial structures. At present, the adoption of cloud-resolving models in operational weather forecasting is imminent, while it is still 5 to 10 years in the future for regional and global climate models, respectively. An increase in compute power by a factor of 100, allows for an increase in horizontal resolution by a factor of 3-4 (as CPU time roughly increases with the fourth power of the spatial resolution). The following developments in our research fields make petaflop-scale hardware configurations a highly desirable tool: - (a) increase of spatial resolution to better represent key physical and dynamical processes in the atmosphere and ocean, - (b) use of ensemble techniques (singular vector, stochastic physics and perturbed physics methodologies) to provide probabilistic estimates of weather, climate and extreme event predictions, - (c) additional interactive and more realistic modules for the processes not (yet) directly resolvable by current models (for instance relating to the interaction of aerosols and clouds). The foregoing developments have important societal benefits in the areas of extreme events and climate change. 2. What is the optimal role of CSCS within the overall strategy and the ideal balance between such a role and other actions? It is certainly beneficial to concentrate national HPC resources into one or a few centers like CSCS, which need to be adequately staffed to provide a high quality service with respect to availability of the machine(s) and support in porting applications to the given platform(s). We believe that the geographical location of a national HPC resource within Switzerland is not a top priority issue, provided the center is well embedded into and well connected with the scientific landscape. It is important, however, that decisions regarding national HPC resources (including management, procurement and allocation to users) are based on scientific principles and the quest for excellence. 3. What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy – that should be considered while planning the acquisition and deployment of new HPCN facilities? July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-39 ETH Board Climate model simulations produce a large amount of data that needs to be stored for a few years. Hence a sufficiently dimensioned data archiving system will be needed alongside the computing platform. For our ALPS project we will archive 185 TByte of data. The amount of data will roughly increase by a factor of 5 when the computing performance increases by a factor of 10. Furthermore facilities should be available for the pre- and postprocessing tasks that do not need a petaflop scale system. In addition of being computationally very demanding our models are very data intensive (i.e. I/O intensive) and our applications need high bandwidth for communication between the individual computing nodes. These properties derive from the high number of degrees of freedom, and from the widespread use of spectral and semi-implicit numerical methods in atmospheric/climate models. Hence our research needs platforms that are well balanced in this respect rather than just a maximum of compute cycles. It is not a priori clear whether our applications will scale well beyond O(1000) processors. Thus the strategy should also account for scientific challenges (e.g. in the area of numerical methods) and manpower that allow addressing these issues, and thus go beyond mere optimization. 4. What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? In most cases major codes for leading HPC systems are the results of national or international collaborations. In our case, we are using regional climate and atmospheric models that share the dynamical core with the weather forecasting model of MeteoSwiss, and we are closely collaborating with leading international institutions (e.g. MPI Hamburg, ECMWF Reading) regarding global models. The future HPCN strategy should encourage such collaborations. For instance, it is beneficial for our institute and MeteoSwiss if these applications can be run on the same platform, such that we can mutually exchange our developments. 5. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? The governing structure of major HPC facilities should entail a strong bottom-up component. The investments only make sense if they fit the needs of the users. There are many international examples of successful HPC centers, which demonstrate how to efficiently organize this process. Again, the decision structure should follow the quest for scientific excellence. The target applications for the new HPCN initiative should be considered in an early stage of the platform selection process, so that an adequate list of requirements can be established. Some of the applications may have requirements that are not well represented by publicly available benchmark suites. Thus selected research groups should be involved in the benchmarking process. 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc…)? On the level of degrees (Bachelor, Master, PhD), we believe that the existing universities will take care of these tasks, and they should also involve national HPC centers into the build-up of a strong educational programme. In addition, the centers should establish the necessary expertise on how to develop petaflop scale applications and transfer this knowledge to the research groups by seminars, workshops, training courses, etc. Much of the research in the HPC area is conducted by individuals that have their primary education in some other field of science and by staff which had their education some time ago. The continued education of these individuals requires particular consideration. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-40 ETH Board 7. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? 8. How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? There are many examples (in Switzerland and abroad) that demonstrate how the transfer of research codes to industry and/or national administration has provided a relevant return on an economic and/or national level. This includes many applications in numerical weather prediction models that to a significant extent rely on university-based research. In our specific area of research international coordination is lead by specific EU programmes (e.g. ENSEMBELS, CECILIA) and by international organizations (e.g. IPCC/UNEP, WMO, ECMWF). The exploitation of this knowledge for Switzerland is only feasible if the respective expertise is available within our country. And this expertise can only be built if suitable HPC resources are available. Zürich, 2006/12/06 Ch. Schär, U. Lohmann, D. Lüthi, S. Seneviratne, M. Wild, S. Ferrachat Institute for Atmospheric and Climate Science Universitätsstr. 16 8092 Zürich July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-41 ETH Board Memorandum Date: From: Phone: Local: e-mail: November 28, 2006 Valeri Markushin +41-056-310-3657 valeri.markushin@psi.ch To: Prof. Ralph A. Eichler ralph.eichler@psi.ch Cc: Dr. Kurt Baltensperger kurt.baltensperger@ethrat.ch Swiss National Strategic Plan for High Performance Computing and Networking V. Markushin, PSI By extrapolating the current dynamics of the Top500 HPC facilities to 2007 - 2011 (Fig.1) one can expect that the commodity HPC machines will hardly exceed the 0.1 petaflop (105GFlops) level by 2011, while the aggregated performance of Swiss based HPC machines will be on a petaflop scale. At the same time, a successful petaflop machine, which can deliver Rmax > 1 PFlops as a single resource, must be a mission driven project and a joint venture with hardware and software vendors. It is unlikely that such a petaflop machine can be used cost efficiently by a broad scientific community in Switzerland in 2007 - 2011. Given the primary goal to strengthen the competitiveness of the Swiss based researchers, the increasing role of computational modelling and simulation, and the foreseen technological progress, one can expect that the best return on investment will be reached with a multi-tier set of HPC facilities with the aggregated performance on a petaflop scale. The multi-tier setup, with the top level machine(s) in the world Top50, will be able to provide the Swiss research and academic institutions with advanced high-capability facilities that are too big or expensive for most national centres. The optimal role of CSCS would be to provide leading edge machines of different architectures with a high impact on the future development of HPC and to lead the HPC market and technology research. A few high-capability machines above the commodity level must be hosted at several national centres to ensure that excellent HPC expertise is readily and directly available to the users and to stimulate long lasting collaborations between HPC users and HPC experts. With the scalability built-in by design, the national multi-tier HPC facilities should provide the possibility of low-cost migration of scalable applications to the top level facilities and efficient post-processing at lower levels, with the issues of resource management, networking, data storage, and data flow properly taken into account. The computing on a petaflop scale is expected to bring in new computational paradigms (high-productivity computing, adaptive supercomputing) since most traditional July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-42 ETH Board -2- computational frameworks cannot scale well to hundreds of thousands of processing elements. It is absolutely essential to provide the users with new programming methods and tools that are necessary for utilization of the new hardware. The universities must keep up with the HPC trends and prepare new generations of users to be "HPC ready". While a teraflop machine can be built as a pile of PCs, it would be a grave mistake to assume that a petaflop scale computing amounts to a pile of HPC commodity clusters. The key mistake to be avoided is the repetition of mistakes, therefore a transparency and a good coordination of the planning and management are essential on the national scale. While planning and deploying the new HPC facilities, one must ensure that there is a close collaboration with the vendor for the whole lifetime of the facility. It would be desirable to implement business-like risk management and introduce independent HPC auditors at key stages of the contract procedures (e.g. acceptance tests). A possibility of joint ventures with Swiss industry, that can be used to scale up "risk free" proven HPC solutions, should be investigated. Fig.1. The dynamics of the aggregated performance Rtotal of the Top500 machines vs. the individual Rmax benchmarks in 2001 - 2007 (left) and the position of the Swiss HPC machines in the Top500 in November 2006 (right). The data are from http://www.top500.org/. SNSP_HPCN_VM-200611.odt July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-43 ETH Board Input Markus Meuwly Dear Manuel Thank you for your invitation to contribute to the SNSP-HPCN. Unfortunately, I will be travelling when the Open Day takes place on December 6th. Please find in the following my views on the points which apply to our case that were raised in your document. Ad 2: The CSCS should provide both, high performance computing (service) and concrete assistance/guidance to use and coordinate these computations. The CSCS should secure Switzerland’s standing in the computationally oriented research that is carried out at our industrial and academic institutions. This includes both, hardware acquisition and maintenance, and education to optimally use these resources. The latter point concerns the continuing education of the users. Ad 4: It would be helpful to run thematic workshops on HPCN at a level suitable to students/knowledgeable non-(yet) experts. As no institution – except maybe the ETH’s – have the HPCN facilities and knowledge right at their place, it is difficult for the majority of interested students to become familiar with HPCN. If both, academic and industrial individuals would be prepared to run such workshops, students could also get insight into the different use of HPCN in the respective environments. Ad 5: One possibility are joint workshops (see Ad 4). Other possibilities are “fellowships” which would allow students/more advanced researchers from Swiss Universities/ETHs to work for some time (1 or 2 months) within CSCS, i.e. directly in Manno. Ad 6: It might be of interest to have funds which are provided on a competitive basis for software development. A committee would need to define key areas (for Switzerland and for both, academia and industry) for which software development is essential and Swiss groups (academic and industrial) should then compete for these funds. I hope these comments are of some use. Sincerely Markus ---------------------------------------------------------------This message was sent using IMP, the Internet Messaging Program. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-44 Swiss Institute of Bioinformatics: Molecular Modeling Group ETH Board July 4, 2007 Research interests: ● protein-protein interactions ● protein-DNA interactions ● protein-small molecules interactions Computing needs: SNSP-HPCN | Final Report | Appendix G ● High interconnect (Mainframe) ● Classical Molecular Dynamics simulations ● integration step 1fs ● typical biological timescales: ns-ms ● number of atoms: >100’000 ● Low interconnect (Grid) ● Drug design applications ● Other embarrassingly parallel problems page G-45 Michielin ETH Board Input Henrik Nordborg -----Original Message----From: henrik.nordborg@ch.abb.com [mailto:henrik.nordborg@ch.abb.com] Sent: Wednesday, November 29, 2006 6:12 PM To: Reinhard Leszek (F&W); Baltensperger Kurt Subject: RE: Swiss National Strategic Plan for High Performance Computing and Networking, SNSP-HPCN Dear Mr Reinhard, Dear Mr Baltensperger, here are the answers to the eight questions from the point of view of ABB. Our perspective is probably different from the one of other participants, as we are currently not major users of high-performance computing. Furthermore, our focus is on applied simulations rather than exploring the potential of HPCN. Nevertheless, I believe some input from the "applied industrial" side can be quite important and I have answered the questions I considered important/relevant to us. ABB has a number of challenging problems that can only be handled using HPCN. Currently, the scientific hurdle to solving these problems is very high. It would be great if a national HPCN initiative could improve the situation. Sincerely, Henrik Nordborg ------------------------------------------------------------------Henrik Nordborg Group Leader - Power Device Simulations ABB Schweiz AG Corporate Research Segelhofstrasse 1 K / Postfach CH-5405 Baden 5 Dättwil phone: +41-(0)58-586 82 39 mobile: +41-(0)79-799 13 46 fax: +41-(0)58-586 73 14 ---------------------------------------------------------------------1. Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the top-performing computing system that Switzerlandshould acquire in 2008-2011? 1. A petaflop-system will probably have little immediate impact on more applied industrial research. The reason is that industrial research uses commercial software to a large extent, and this software is typically not available for the largest and fastest systems. The simulations typically also have the character of parameter studies, and it is therefore more important to be able to run many smaller jobs than one big one. Therefore, ABB would be more interested in flexible multi-purpose machines than a top-notch petaflop system. 2. What balance is the optimal role of CSCS within the overall strategy and the ideal between such a role and other actions? July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-46 ETH Board 2. CSCS should take a very active role in promoting HPCN both to academia and industry. To my experience, availability of a big computer does not automatically lead to people using it efficiently. It is becoming more and more difficult to get optimal performance out of high-performance systems, and smaller research groups and industrial research centers do not have the necessary know-how. 3. What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy ? that should be considered while planning the acquisition and deployment of new HPCN facilities? 3. Again, I would wish flexible solutions, rather than extremely high performance for one specific application. For industrial applications, some consideration should also be given to confidentiality. A national GRID will only be used by industry if it is sufficiently secure. 4. What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? 4. I am not sure that I understand this question, but it seems that "academic contributions" to HPCN can only come from applications of large scale simulations to relevant problems. The key would be to identify some important problems -- from industry or academia -- and ensure that these obtain suffient funding for the HPCN-related part. 5. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific financial credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc?)? 6. Credit points is probably a good idea. In particular, I would like to see creadit points for HPCN projects that also involve industrial research. 7. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? 7. HPCN will play a key role in engineering in the future (cf. "Simulation Based Engineering Science" from NSF, May 2006, http://www.nsf.gov/pubs/reports/sbes_final_report.pdf). As is pointed out in this report, the main obstacle is not lack of computers but rather lack of knowledge and algorithms. Whereas academia to some extent can choose research topics suitable for high-performance computing, industrial research does not have this freedom. It would be a major mistake to just provide the hardware, without a dedicated support team with real customer focus. This is especially true if one wants HPCN to have an impact on industrial research. 8. How will HPCN in Switzerlandimpact and enable the groups to provide national services? scientific application July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-47 Open Day HPCN Strategy, 06.12.06 Finding a HPCN Strategy for Switzerland ETH Board July 4, 2007 1. „Do we need a Petaflop system?“ – This is the wrong question to ask! Peak performance is not an appropriate measure for a HPC system. 2. HPCN strategy: must be driven by the users of HPC systems, including: Universities & ETH domain, MeteoSwiss, Industry. All need to be represented in the project group! SCS: Service Centre for Science 3. The role of CSCS SNSP-HPCN | Final Report | Appendix G ◊ Main service: operation of HPC systems and environment! ◊ Other services: user support (e.g., code optimization, visualization), evaluation and procurement of HPC systems ◊ Continued close contact with HPC users on various levels is indispensable (… in particular during procurements!) page G-48 Dominik Obrist & Leonhard Kleiser Institute of Fluid Dynamics ETH Board Input Artem Oganov Dear Prof. Margaritondo and Prof. Baltensperger, Below are my comments for the Open Day. Unfortunately, I cannot attend the Open Day because of teaching, but I hope that my views could be discussed openly: 1. Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the top-performing computing system that Switzerland should acquire in 2008-2011? ANSWER: Yes, a petaflop system is highly desirable for Switzerland. CSCS is currently oversubscribed by several times. This needs to be urgently corrected already now. Taking into account Moore's law, around 2010 we will need a capacity of about 1 Petaflop. Due to limited resources of CSCS, many research groups do not apply to CSCS for computing time. Greater computing facilities at CSCS will attract more scientists to become users of CSCS, creating a larger pool of competence there. Presence of large-scale highly scalable supercomputers will stimulate researchers to design their algorithms in more modern ways, taking maximum advantage of latest advances in hardware. Most importantly, large-scale facilities will enable the most groundbreaking, scientific projects - thus stimulating scientific progress in Switzerland. 2. What is the optimal role of CSCS within the overall strategy and the ideal balance between such a role and other actions? ANSWER: In my view, CSCS should be the centre point of computational initiatives in Switzerland. However, it should not become the only large computing facility in the country. 3. What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy – that should be considered while planning the acquisition and deployment of new HPCN facilities? --4. What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? --5. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? ANSWER: Present organisation of CSCS within ETH domain seems quite efficient. Maybe CSCS could become a joint venture of the whole ETH Domain, rather than part of only ETHZ. 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc…)? ANSWER: No. This will make the system more bureaucratic - and as a consequence, less efficient and less attractive for scientists to participate it. 7. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? ANSWER: It may be a good idea to establish joint academic-industrial computational projects at CSCS. This could be an additional source of money for CSCS. At the same time it seems logical to avoid defence-related projects in such public facilities as CSCS. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-49 ETH Board 8. How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? --- With my best regards, Artem R. Oganov Vice-chairman, special interest group "Mineralogical Crystallography" http://www.lcm3b.uhp-nancy.fr/ecasig5/ ************************************ Prof. Artem R. Oganov *Senior Researcher and Group Leader, Laboratory of Crystallography, ETH Zurich, Wolfgang-Pauli-Strasse 10, Zurich CH-8093, Switzerland **Adjunct Professor, Geology Department, Moscow State University, 119899 Moscow, Russia Tel.: +41-(0)44-632-3752; Fax: +41-(0)44-632-1133 Personal web-page: http://olivine.ethz.ch/~artem ************************************* RESEARCH NEWS Crystal structure prediction method: http://olivine.ethz.ch/~artem/USPEX.html ************************************* July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-50 ETH Board Dr. Daniele Passerone Group Leader Theory and atomistic simulation of materials Department nanotech@surfaces Empa, Dübendorf Statement for the HPCN Open Day 2006 Empa has recently started an activity in computational science, to provide a theoretical and computational service group for the experimental research activity taking place within the institution. Within this initiative, a theoretical group has been started. Jointly, a small Linux cluster has been purchased. Initially, the cluster (Dell, Intel Woodcrest-based) is made of 20 biprocessor nodes, with a total of 80 CPU cores and 160 GB of RAM. The interest in this initiative at Empa is growing, and that will bring to the doubling of the computational cluster within 18 months. My group will act both as service group for experimentalists wishing to model specific materials science related phenomena, and as coordination center for computational scientists present in different departments at Empa/Eawag. In this framework, these are some answers to specific questions: 1) Should Switzerland acquire a Petaflop system? My answer is positive, within a coordinated national strategy. Materials science, medical innovation and chemical and biological research would profit from the presence of such a resource. The predictive power of computer simulation has increased enormously in the last years, and “real world” applications coming from this branch of theoretical research start to be available for production. The recent experience at Empa, for example, has shown that the development of novel materials for industry can be drastically improved with the aid of specific fundamental research. Realistic modelling of innovative materials and drugs requires both novel algorithms and theories and massive computational resources. The latter should be surrounded by an adequate infrastructure (see below). The former needs instead the development of computational science groups that should concentrate on the scientific part of the projects, with the assistance of local and national (CSCS) teams of experts for the implementation of their algorithms within a particular architecture. 2) What is the role of CSCS within the overall strategy? Handling a resource like a petaflop system has no meaning outside a supercomputing center, due to the extreme complexity of its administration. Moreover, a keyrole is played by competence transfer to smaller and local HPC centers. 3) What are the technical issues - such as networking, data broadcasting and storage, data analysis, protection and redundancy – that should be considered while planning the acquisition and deployment of new HPCN facilities? Networking and storage are interconnected problems. The storage is one of the most important issues and should be (at least on a large scale) centralized at CSCS. Networking has to be strongly enhanced to access the storage and transfer huge data from large scale calculations in a more efficient and direct way. Redundancy of electrical power and redundancy of storage represent two important technical issues. 4) What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? psd127, 03.12.2006 C:\Documents and Settings\balkur\Desktop\Feedbacks HPCN\Passerone Daniele 061201 Empa_statement.doc July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-51 1/2 ETH Board Establishing an equilibrium between optimization of resources, and subsequent clustering of local needs (e.g., different departments at Empa refer for small and medium sized projects to the same central Empa HPC cluster), and harmonization with the supercomputing resources at CSCS. 5. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? Reference persons in each academic institution should provide to the interested local computational scientists: 1) Guidelines for choosing the appropriate resources where to perform their calculations, using the information obtained from the competence center and with continuous exchange with the CSCS crew. 2) Guidelines for an effective and reasonable submission of large and small scale projects to CSCS. 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc…)? Yes. These credit points are presently missing – typically, SNF projects cannot include consumables for computers etc. This would stimulate the growth of local HPCN and the interconnection of smaller resources with the protocols of grid computing. Very often, acquiring a HPC cluster represents a sizeable effort for an academic group with limited financial resources. On the other hand, a strategy including a central Petaflop computer should consider as extremely useful the financial support of scattered computational clusters in the Swiss academic institutions. The optimization of such computational resources should happen at a local level, so that these can act as nucleation seeds for computational science activities directly connected to the research of interest in the hosting institution. 7. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? As reminded at point 1, the industrial competitiveness would enormously profit from the enhancement of computational resources. A key factor of success in this strategy should be an active collaboration between industries and academic institutions, with the establishment of application-oriented projects with industrial financial support. Although the public funding should always remain dominant in order to grant the independence of fundamental research, the industries should be aware that computational science can provide an innovative path of research and development to achieve the company’s goals. Dissemination of methods and results and transfer of competence from CSCS and local HPCN to the industries should help to tighten the contacts between the two worlds. The formation of an interconnected structure able to harmonize computer science expertise and computational science groups in the whole Switzerland will allow the Country to act as a stronger referent for international projects. The participation to European programs (like 7th Frame Program) has to face strong competition from other European institutions. The recent establishment of the most powerful supercomputer in Europe in Barcelona is an example of the urgency of keeping the edge in this framework. 8. How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? HPCN with its infrastructure will help in tightening contacts among scientific application groups in Switzerland. As local computational clusters can serve to coordinate research effort within a single institution, the presence of an invasive supercomputer center with enhanced computing capabilities as a petaflop system can catalyse joint efforts of national interest: for example, Biologically oriented grid computing, meteorology forecast services, national initiative in the development of materials with novel electronic properties (MANEP) have obtained important results in the last years. The development of a national HPCN program offers the unique possibility to bring together computational science entities that barely know each others. For this reason I stress again the fact that the purchase of a petaflop system without an adequate national and sizeable support to local computational resources could represent a partially wasted investment. psd127, 03.12.2006 C:\Documents and Settings\balkur\Desktop\Feedbacks HPCN\Passerone Daniele 061201 Empa_statement.doc July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-52 2/2 ETH Board Input from Walter Pletscher for OPEN DAY Dear Professor Giorgio Margaritondo, in agreement with our Lab Director Matthias Kaiserswerth (on business trip) , find attached the statement from the IBM Zurich Research Laboratory as respond to the questionnaire we have received on November 29, 2006 from Board of the Swiss Federal Institutes of Technology member Kurt Baltensperger. Our response is a strong supporting statement to the ongoing initiative. We have compiled the IBM Research Laboratory view as follow: There’s growing evidence that most developed economies, including Switzerland, will need and are planning to invest in leading-edge, capability computing platforms to maintain their competitive advantage in the areas of scientific research and education, technology innovation, and industrial applications. Such a platform will enable key industries and academia in Switzerland to adapt through innovation to the rapid changes in the global business environment, and to further enhance its human capital: skills, ideas and the capacity to explore new frontiers. The unique scientific insights and engineering capabilities it will deliver are critical to sustaining Switzerland’s competitiveness in the global economy. It will benefit large enterprises (pharmaceutical, financial markets, insurance and banking, electronics among others) as well small and medium businesses (SMB or KMU Mittelstand). Deploying such a platform can only be achieved by joining forces. A single Swiss institution may be able to build a powerful infrastructure, but its scale would still remain small compared to national initiatives under way in other countries. Spreading Switzerland’s HPC investments and infrastructures across two or more platforms would be sub-optimal, both in terms of scientific value and overhead costs. The HPCN initiative is strongly supported by IBM Research . This approach intends to reach beyond a single company or academic department’s interest, to bridge across language and industry barriers, and represents an opportunity for Switzerland to take a lead role in the next generation of computational science. Best regards / Meilleures salutations / Mit freundlichen Gruessen for Matthias Kaiserswerth, Director IBM Zurich Research Laboratory Walter Pletscher Manager Industry Solutions Lab (ISL) IBM Research GmbH Zurich Research Laboratory Saeumerstrasse 4 8803 Rueschlikon Switzerland Dept. YI8 Mail: plw@zurich.ibm.com Phone: +41-44-724-8713 Fax: +41-44-724-8710 Mobile: +41-79-403-1355 July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-53 ETH Board From: Tel: E-mail: Data: Prof. Giambattista Ravano +41-(058)-666-6552 giambattista.ravano@supsi.ch 30.11.2006 Swiss National Strategic Plan for HPCN - Answer to ETH Board questions Thank you for being involved in the Strategic plan for HPCN. Our statements apply to the points 2 through 6. We believe the acquisition of a computational system of the highest performance could be a major step forward for the scientific infrastructure of Switzerland. Such systems could be realized with various technologies (possibly more then one should be investigated and used) and, through an efficient and reliable network, be utilized from all the Swiss scientific community. CSCS role should move from pure support of operation to a center of excellence in software development for HPCN applications and systems operations. Other actions or centers in Switzerland have to concentrate on specific applications domain of HPCN like bioinformatics, physical chemistry, nano physics, materials, etc.. Many technical issues are important and could move forward the Swiss scientific excellence in the HPCN area. Data broadcasting and storage together with software methodologies could be a useful specialization area. The academic contributions to HPCN in Switzerland should be on software more then hardware and on applications and systems software more then pure operations. CSCS and other HPCN centers should be linked each to one or more university or university like institution, preferably to Computer Science or Engineering Faculty, ensuring in this way their presence in the scientific community. The specialization of the HPCN center should reflect the scientific specialization of the host university. The presence of USI and SUPSI in Cantone Ticino, where CSCS is based, reinforces the motivation on having a software methodologies and systems operations specialization. Credit points in this technological area could be offered in Master programs. Of course this Masters should have a more general goal and offering a complete know how about software engineering, or systems management. Specialized Master degree’s on topics handled in HPCN centers could also be offered in future, mainly where the application is devoted to a specific scientific field. Giambattista Ravano Direttore Dipartimento Tecnologie Innovative Scuola universitaria professionale della Svizzera italiana July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-54 ETH Board Input Swiss Re From: Andrea_Schlapbach@swissre.com [mailto:Andrea_Schlapbach@swissre.com] Sent: Tuesday, December 05, 2006 5:10 PM To: Reinhard Leszek (F&W) Cc: Christoph_Locher@swissre.com; Markus_Spuehler@swissre.com; Robert_Marti@swissre.com; Manfred_Kaufmann@swissre.com; Heinrich_Waldhier@swissre.com; Bernd_Goetz@swissre.com Subject: Swiss National Strategic Plan for High Performance Computing and Networking: Swiss Re's feedback Dear Sir You have asked Swiss Re to provide a feedback on the "Swiss National Strategic Plan for High Performance Computing and Networking". As announced to you last week, here is the feedback we have agreed on: Swiss Re is the world’s leading and most diversified global reinsurance company. The company operates through offices in over 30 countries. Founded in Zurich, Switzerland, in 1863, Swiss Re offers financial services products that enable risk-taking essential to enterprises as well as the society at large. The company’s traditional reinsurance products and services in property and casualty as well as in the life and health sector are complemented by insurance-based corporate finance solutions and supplementary services for comprehensive risk management. Risk management is an important activity for the financial services industry and involves both qualitative and quantitative assessments. Such quantitative assessments require the ability to adequately assess the potential impact of adverse situations with minimal delay. It includes the rapid risk assessment prior to signing an insurance contract, the correct pricing, the ongoing monitoring of limits for adequate solvency and liquidity, the tactical and strategical steering of the company and its units as well as providing regulators and rating agencies with up-to-date, accurate risk information. While risk management methods such as value-at-risk metrics for investment banks are already well established, a reinsurance company faces a multitude of heterogeneous risks such as credit, pandemic risks, liability, credit, inflation, terrorism and natural catastrophes. Looking into each of these disciplines in a quantitative manner involves a wide range of scenarios - even some quite unlikely ones - using product-specific risk models. Moreover, the separate results must be integrated while accounting for diversification effects between these different products. All of this is a demanding undertaking from a data, process and computational view. To do so Swiss Re currently maintains three major distributed computing infrastructures for operational use: - Integrated Risk Management assesses the company's overall risk exposure, used for steering the company, ensuring an appropriate rating and regulatory compliance from a solvency point-of-view, allocating risk-adjusted cost to all business units and for limit setting. - Cat Peril is assessing the specific risks caused by natural catastrophes such as earthquakes, tropical-, wind- and hail-storms as well as flooding. Their results are used by Integrated Risk Management. Cat Peril supports underwriters for appropriate pricing for contracts exposed to natural hazards. Cat Peril has an ongoing intense collaboration with University of Zurich on HPC optimisation. - Capital Management and Advisory does typical banking risk management for the company's trading desk, such as pricing and valuation modelling for various vanilla and exotic financial instruments such as credit derivatives, equity derivatives, interest rate derivatives etc. The needs for calculational performance for these tasks have continuously grown and are expected to grow even further, impacting the infrastructure required. As a result we are continuously assessing possibilities how to serve these needs also from an infrastructural point of view, especially when the cost in a strategic timeframe can not be neglected. Regarding a "Swiss National Strategic Plan for High Performance Computing and Networking" we would like to make the following comments: July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-55 ETH Board - HPC is a technology of importance for both R&D and operational needs. We share the view that Switzerland's institutions should assign a high importance to this topic, especially on the educational side. As the cost to operate such a top-end infrastructure and to build-up the corresponding skills is very high, there is a clear need to align efforts between different institutions inside and outside the country. - The financial services industry's needs on HPC exist. These needs are primarily of operational and not R&D nature. While it is possible that increasing HPC needs lead to acquiring computing performance services from a 3rd party, it is rather unlikely that such an operational service is easily shared with academia as the service level needs of commercial enterprises are different, and potentially much higher outside of the pure performance metric. Furthermore, top-end HPC infrastructures often require very specific application architectures, leading to a footprint which can be assessed as being too high. - The insurance industry will continue to have a rather intense R&D involvement as risk management - in contrast to investment banking - is still a very evolving discipline. For this reason, active collaboration with academia already exists and needs to continue, i.e. these skills must be available. Such skills include risk modelling, implementation and HPC. With these comments I remain. Please let us know the collected feedback from industry and the proposed next steps with regard to the "Swiss National Strategic Plan for High Performance Computing and Networking". Unfortunately, we will not be able to participate at tomorrow's event in Bern. Kind regards Andrea Schlapbach Andrea Schlapbach | Member of Senior Management | Head of Integrated Risk Systems Swiss Reinsurance Company | Mythenquai 50/60, 8022 Zurich, Switzerland Direct: +41 43 285 5193 Fax: +41 43 282 5193 Mobile: +41 79 472 8252 E-mail: andrea_schlapbach@swissre.com http://www.swissre.com This e-mail, including attachments, is intended for the person(s) or company named and may contain confidential and/or legally privileged information. Unauthorized disclosure, copying or use of this information may be unlawful and is prohibited. If you are not the intended recipient, please delete this message and notify the sender July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-56 ETH Board Input Beat Schmid Statement from the University of St. Gallen Demand. In economics and finance we see a trend to more and more complex models. In finance very large models in fields such as asset allocation optimization, risk management and others are state of the art today and will increase in the future. In economics, beside the already established quantitative disciplines, we see an new emerging field in experimental economics, where e.g. institutional designs, or protocols for electronic markets, are tested for their behavior. Such developments may be compared with computational physics or computational chemistry, and it may be expected, that they will see a similar growth in the future, as these disciplines have seen in the last years. As university with the goal to belong to the leading schools we want to offer our faculty and students the best means for their research work. Therefore, we wish to have - availability of computing power in the top segment, and - easy accessibility of it for researchers of all academic levels, even students. Technical solution. We do not propose a specific solution, as Petaflops machine, or a grid of similar computing power. We leave such judgement to other schools, closer to technology. Institutional aspects. Here again we do not propose a specific solution, as CSCS, or SWITCH, or a combination of the two. We are interested in the availability and accessibility, while organizational aspects may be of more concern for other universities. ******************************************* Prof. Dr. Beat F. Schmid Institute for Media and Communication Management University of St. Gallen Blumenbergplatz 9 CH-9000 St.Gallen Tel. :+41 (071) 224 2770 Fax : +41 (071) 224 2771 e-mail: beat.schmid@unisg.ch www.mcm.unisg.ch July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-57 ETH Board IBM Research GmbH – Zurich Research Laboratory Respond to Swiss National Strategy for High Performance Computing and Networking 1. Should Switzerland acquire a petaflop-level system? If so, what are the technical and scientific justifications in terms of major national projects? If not, what is the topperforming computer system that Switzerland should acquire in 2008-2011? Yes, it is in Switzerland’s best interest to acquire a petaflop-level system in the 20082011 time frame. Large-scale simulations have become an essential tool in many disciplines, many of which are national priorities in Switzerland: systems biology and molecular biology for pharmaceuticals and healthcare; nanoelectronics for the postCMOS era; materials science for nanotechnology; mechanics for both fine machining and micro-/nano mechanics; particle physics (CERN) and energy (ITER); catastrophic risk management for both safety and insurance; financial simulations for banking; and in the future new services concepts for business. In these fields, physical tests / experiments must increasingly be augmented by simulations for practical and cost reasons. A PFlop machine would make possible full, accurate and comprehensive simulations on real (as opposed to model) systems, making in-silico design a reality. It is not just a question of “capability” computing, but also “capacity” computing, for the trends in computational science are such that Switzerland will need to have a computing facility to run perhaps 20 projects in different fields each on the scale of 50 teraflops. With increasing global competition there will be pressure to maintain and expand the knowledge and skill base in computational sciences. Similar initiatives are on-going in the USA, Japan and Europe. 2. What is the optimal role of CSCS within the overall strategy and the ideal balance between such role and other actions? The optimal role of a HPC center with a petaflop-scale installation is not just to provide computing cycles, but also to attract global talent, as well as support and drive innovation in scale out applications. 3. What are the technical issues – such as networking, data broadcasting and storage, data analysis, protection and redundancy – that should be considered while planning the acquisition and deployment of new HPCN facilities? Greatest issue with next generation HPC machines will be power consumption. Another the programming model and support environment (e.g. porting applications). Using standard solutions to exchange, store, retrieve and backup data, is one aspect. Emphasis should be given to real time data analysis and visualization. Requirement: Standardization of data formats. A proliferation of formats will significantly reduce the efficiency and/or possibility of necessary cooperation and lead to exploding cost. The architecture of the system itself should allow adequate scalability over a wide range of processor counts and be partitionable. Although the central computational power should be located in one place, a high-performance distributed data infrastructure for sharing data is important. The architecture should be based on experiences in other European HPC projects like the DEISA grid project. SWITCH would play an essential role. 4. What are the most effective measures to enhance the academic contributions to HPCN in Switzerland? IBM_HPCN_Bern-V3.doc July 4, 2007 Page 1 SNSP-HPCN | Final Report | Appendix G 06.12.2006 page G-58 ETH Board IBM Research GmbH – Zurich Research Laboratory Respond to Swiss National Strategy for High Performance Computing and Networking Effective measures would be organization of workshops on application scale-out, appropriate showcases of successful projects and introduction of "massive parallelism techniques" in academic curricula. 5. How should CSCS and the other HPCN centers be linked with the scientific community and leadership? Linkage can be fostered through regular exchange of "HPCN practitioners" within country and abroad. HPC centers should be more than simply a source of “computing cycles”, but rather a source of competencies, supported with a good scholar program. 6. Should the ETH domain, Swiss universities and/or the universities of applied sciences establish specific credit points in the area of HPCN (Operations, GRID computing, software development and maintenance, etc…)? There is a fundamental difference between the skills needed to operate and those needed to use a HPC infrastructure. The operational side is lower skilled and less suited for academic credit points, while the "run/program/interpret" is significantly higher skilled and very suited for academic credit points. It might well be envisioned that almost every curriculum of an education requires lectures / exercises in programming HPC systems. 7. How will the HPCN strategy impact the national security and industrial competitiveness of Switzerland? What are the key mistakes that must be avoided? There is a direct correlation between HPC capabilities and innovation and economic competitiveness. In the Swiss economy, this means pharmaceutical, insurance, banking, and technology in industrial sectors. Access should be granted to both large and small businesses as well as academia. Investment in such a machine will pay itself back, by providing solutions to otherwise intractable problems and permitting a quicker time to market, ultimately increasing Switzerland’s competitiveness on a global scale. A key mistake to avoid is fragmentation of such a large investment across multiple installations. The benefit of having one single installation has to be seen in context of potential applications where scaling is a mandatory requirement. More importantly though, multiple sites would substantially increase implementation and operation costs. It is also important to recognize that a HPC center is more than just the hardware - having competent people with the right skills is essential. Porting and redesign of applications for a massively parallel computer will require talent and new approaches. 8. How will HPCN in Switzerland impact and enable the scientific application groups to provide national services? Each group represents key competencies in specific areas. Access to such a facility as HPCN will keep them at the leading edge of science and technology. It will enable them to exploit synergies among them. IBM_HPCN_Bern-V3.doc July 4, 2007 Page 2 SNSP-HPCN | Final Report | Appendix G 06.12.2006 page G-59 ETH Board Input Thomas Smith, Swiss Tropical Institute Dear Manuel Peitsch, We are writing on behalf of the biostatistics and modeling group of the Swiss Tropical Institute, which has several computationally intensive projects in the areas of (i) stochastic modeling of malaria dynamics and in (ii) geostatistical analysis and mapping of pathogen distributions in Africa. We will be represented at the meeting next week by Nicolas Maire who is our chief software developer. The group comprises currently 6 scientists together with a number of postgraduate students and has an annual budget of about CHF1 million. We are using a volunteer computing platform for our stochastic modeling work. This has performance in the range of several teraflops, and could easily be increased. We are exploring different options for the best approach for the computationally intensive problems in the geostatistical work (which uses Markov Chain Monte Carlo algorithms for fitting models, and is less easy to adapt to highly parallel systems). We think a petaflop-level system is indeed likely to be of strategic importance for Switzerland but are not sure whether it will provide the answer for our own applications. For users like ourselves, an important consideration is to have software engineering skills accessible for porting mid-level projects to new systems, to fully exploit the architecture. Many academic projects do not have enough resources to employ full-time software engineers, and have problems communicating their needs. Volunteer computing is likely to become increasingly accessible, so it will be important to avoid making major investments in hardware that will end up competing with this on applications that lend themselves to distribution over the internet to volunteers (this is a highly cost-effective alternative for applications where it is practical). In terms of national services, possible applications in our field are in disease mapping (though currently we are not working on Swiss data) and in simulating disease transmission networks, where some work is in progress in collaboration with ETH. Regards, Tom Smith & Nicolas Maire Tom Smith Swiss Tropical Institute Department of Public Health & Epidemiology Socinstrasse 57 P.O . Box CH-4002 Basel Switzerland Tel 61-284 8273 Fax 61-271 7951 email: Thomas-A.Smith@unibas.ch http://www.sti.ch/en/research/biostatistics.html July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-60 ETH Board Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Swiss National Strategic Plan for HPCN Meteoswiss vision on some key questions MeteoSwiss is calculating time critical numerical weather prediction at CSCS to forecast weather in complex topography, and to provide input for environmental models (e.g. pollutant dispersion, flood forecast). National security, industrial applications and R&D benefit from this system. Its core is a community model currently developed by an international consortium of meteorological services and universities. Based on more than 15 years experience in HPCN, MeteoSwiss offers the following vision on some of the key questions of the SNSP-HPCN. 1. Two aspects are driving the need for additional computing capacity: the necessity to calculate the evolution of the probability distribution function to better forecast severe weather, and an increase of mesh resolution and model complexity to improve the quality of forecast in complex terrain. We will be ready to tackle these issues towards the end of the period 2008-2011. The first goal is achieved by the so-called ensemble method requiring the calculation of O(10-100) forecast, the second goal implies refining the mesh from the current 2 km to 1 km or less. The increased focus on local phenomena entails a higher time criticality for the whole system, which will also be used as a nowcasting tool. The realization of these objectives is only possible if Switzerland acquires a petafloplevel system towards the end of the period 2008-2011. Meteo applications require the highest possible computing power during short fixed time slots; thus, it makes a lot of economic sense to share the needed HPCN resources with other users. Moreover, the large amount of data produced by the forecasting system of MeteoSwiss is used by many other groups which benefit from a common platform. The primary role of the CSCS should be to provide this common high capability platform. More generally, to maximize synergies and GFlops per Swiss Franc, we do think that it makes sense to have a single Swiss HPC centre, caring for the needs of all challenging national projects. MeteoSwiss would not have been able to build the very demanding system it is now using without the excellent support of the CSCS. The second most important role of the CSCS should be to support the users for all IT related aspects of the systems they are deploying at CSCS. This also includes participation in the design phase, help for debugging and optimization, support for migration, and should not be restricted to the HPC applications. We envisage the CSCS as a highly qualified service centre, which takes the necessary steps to enlarge its competences in HPCN, but whose main focus is the needs of its clients. Some technical issues which should be considered while planning new HPC facilities are: (a) capable job sharing and scheduling to handle users with very different needs, including fix schedule real-time applications and on-demand high priority applications. (b) Stability and maturity to ensure high reliability of critical applications, such those related to natural hazard. (c) Tight integration with other CSCS resources, such as archiving and post-processing platform. (d) Minimization of migration effort for the users concerned. SNSP-HPCN | Final Report | Appendix G page G-61 2. 3. July 4, 2007 ETH Board 7. A necessary condition to improve the quality of a numerical weather prediction system is the access to larger computing resources. The most promising path for MeteoSwiss to significantly improve the quality of its numerical products is through the realisation of SNSP-HPCN. In this sense, all clients of MeteoSwiss will be impacted by the success or failure of this plan: the national security (natural and industrial hazards), the cantons (air quality), the industry (energy sector, reinsurances), etc. The key mistakes to avoid are: (a) frequent strategic changes, (b) deficient communication, loss of transparency in the decision process, (c) by-passing the experience of end users while planning the acquisition of new HPC facilities. In relation with the latter point, a central element of any procurement should be a representative set of users’ benchmarks; different applications require different HPC architectures. No specific answers are given to the questions 4, 5, 6, and 8, MeteoSwiss being not part of Academia. However, MeteoSwiss contributes to science and education through common projects involving Diploma and PhD students, and by providing a wealth of model data to academic groups. 1. Dezember 2006 / P. Binder, Ph. Steiner, J.-M. Bettems 2/2 July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-62 ETH Board Statement to the project group of the SNSP-HPCN Prof. Ulrich Straumann, Physik-Institut, Universit¨t Z¨rich a u The Swiss high energy physics (HEP) community is strongly involved in the development and construction of the LHC experiments at CERN and plans to contribute to the exploitation of the data, which will become available from 2008 onwards. LHC will produce about 10 PetaBytes of data per year, which requires a worldwide effort for their analysis. The international HEP community had decided some years ago to rely on GRID technology, based on a large number of relatively simple and cheap CPU and storage devices. In the present prototype phase 177 computing centers are participating actively in the LCG Grid worldwide, providing a total federated resource of 27759 CPUs and 43776 TB of disk space, which is used routinely for simulated data analysis. The Swiss HEP community is building up a common analysis center (of type Tier-2 in the language of the LCG) at the CSCS. Research groups from the universities of Berne, Geneva, Zurich, the ETHZ and the EPFL are involved in this effort, the necessary funds were so far acquired by a common research request to the SBF through the SNF. In addition some contributions from the Universities, the ETH and the PSI are available. The crew in Manno supports us in setting up the system, and provides power, cooling and 24/7 operator coverage. As of today, Switzerland contributes only below the percent level to the worldwide LCG. Until 2009 we plan to ramp-up the LCG resources at CSCS to 2300 kSi2000 CPU power and 787 TB of disk space. Comments to some of the addressed points: 1,) Acquire a petaflop-level system? Do not buy a big system, which is supposed to serve everybody. The user acceptance is unpredictable, while the performance will be outdated soon. Instead invest in several small systems which are adapted to single projects or to a few projects with similar demands, and which can easily be scaled-up, when required. 3. Technical issues: Most importantly CSCS should provide technical expertise, network connections, space, power, cooling and 24/7 operator coverage. The network connection to Zurich, Geneva and Bern is satisfactory for the HEP needs, also for the foreseeable future, assuming no large data flow from other projects. For other research communities longterm, safe storage of research data (e.g. satellite data for geographical research) is important as well. 4.) and 5.) Link to scientific community:. Researchers will always use the cheapest computing power, which is available on shortest notice with as little administrative overhead as possible. From experience, centrally planned HPC centers do not look favorable in this respect. Thus, the potential users have to be involved into the 1 July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-63 ETH Board planning and organization of any computing project right from the beginning. The users should organize themselves for distributing the available computing resources. External administrative overhead should be avoided altogether. (see also 7b) 6. Finances: There should be a funding system, where research groups can request funds for their specific computing needs, including hardware and manpower. One could require, that these funds must be invested in the CSCS Manno. 7. Key mistakes: a) As for any other research infrastructure, only concrete projects and and well defined scientific research plans can justify an investment into a big computer. b) Avoid to have too many groups using the same system. This generates administrative overheads, which need resources both technical and manpower, and make the system unattractive to the users. c) Do not plan computing power without the necessary infrastructure and technical manpower (unlimited work contracts are essential) to run them reliably. . December 1, 2006 2 July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-64 ETH Board EIDGENÖSSISCHE TECHNISCHE HOCHSCHULE LAUSANNE POLITECNICO FEDERALE DI LOSANNA SWISS FEDERAL INSTITUTE OF TECHNOLOGY LAUSANNE CENTRE DE RECHERCHES EN PHYSIQUE DES PLASMAS Association EURATOM - Confédération Suisse Bâtiment PPB - Station 13 - CH - 1015 LAUSANNE TELEPHONE: +41 21 - 693 34 87 TELEFAX: +41 21 - 693 51 76 ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE Prof. G. Margaritondo VPAA EPFL V/réf. Dear Giorgio N/réf. Lausanne, 04 December 2006 I am pleased to submit to you our opinion about the initiative concerning the HPCN. The Centre de Recherches en Physique des Plasmas (CRPP) strongly supports this initiative and I also would like to offer a few other comments. • The CRPP is the Swiss Centre of excellence for fusion research and, therefore, has a national mandate as the executive branch of the Association Euratom-Swiss Confederation. Our theory group has been involved in HPC for decades. It has a leading international role in the numerical modelling of fusion plasma physics phenomena and in the field of material science. It is now one of the active partners in the Blue Gene Project at EPFL. The most demanding computing needs of the CRPP are (and will be) in the areas of first-principle based simulations of turbulence and of kinetic effects on fluid instabilities in magnetic confinement systems. When ion turbulence alone is considered, we estimate the need for one full-scale simulation of an ITER-size plasma to be equivalent to 4 BG/L racks (8192 processors) for one week. Extension to include the physics related to electron dynamics is being performed. This extrapolation yields at least an order of magnitude higher requirement. The availability of hundred of Teraflops/s system in the next few years is considered as mandatory for us to maintain our leading role in the fusion community and to prepare for the scientific exploitation of ITER and for the plasma physics and material science required by the step beyond ITER, the demonstration reactor DEMO. A recent study by experts of the European Fusion Community confirms that the scientific needs of the Fusion Programme in the next five years requires a HPC with performances in the order of a few hundred of Teraflop/s. • July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-65 ETH Board CENTRE DE RECHERCHES EN PHYSIQUE DES PLASMAS page 2 • • • If the final decision is to consider a Petaflop/s system, the time scale for its actual implementation should be compatible with the above requirement, namely to have a working system for scientific exploitation during the next few years. The initiative for HPCN should be a long term strategy with means not only for the procurement of hardware but also for building up, in a sustainable way, scientific and technical competencies in laboratories, specially if it is decided to go for Petaflop/s system. The CRPP has for many years applied this strategy, which has proved to be highly successful on the scientific, and academic (training of young researchers) grounds, Our approach is also highly cost effective and efficient from the managerial point of view. Other initiatives, similar to the Swiss HPCN one, are being discussed in the EU for the fusion programme. If a Petaflop/s system is decided for the next five years, it may be advisable to explore possible collaboration with the EU fusion programme and the role that Switzerland can play in this field in Europe. I am also enclosing in Annex our answer to the questionnaire, wherever we feel that we can contribute to the discussion. I am looking forward to the meeting on December 06th and remain Faithfully yours Prof. M. Q. Tran Director of the CRPP Annex: Answers to questionnaire. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-66 ETH Board CENTRE DE RECHERCHES EN PHYSIQUE DES PLASMAS page 3 Answers to questions Contribution from the Centre de Recherches en Physique des Plasmas- EPFL Preliminary remark The priority should concentrate in assessing the computational requirements of scientific projects and endeavours in Switzerland, and then adjusting the HPCN systems to satisfy these needs rather than the other way around. The current emphasis on procurement of massively parallel systems based on peak performance measurements of simple mathematical algorithms fails to recognize that an important number of real applications that involve strong nonlinear and non-local effects achieve a very weak fraction of the peak performance quoted as a result of scalability problems to a large number of processors. The “Petaflop/s system” should be part of a much broader and long term vision about HPC in Switzerland. It can serve as a “flagship” and thus will hopefully attract the best human competences and stimulate the most challenging projects in the country and beyond. However, such a system alone will not be able to serve all HPCN needs, which are diverse and will almost certainly require several different types of computer architecture to be fulfilled. The recent evolution of computer architectures points to several crucial issues such as memory access and interconnect characteristics in order to assess the performance on real applications: “no single number can reflect overall performance” [Jack Dongarra, UTK/ORNL, www.top500.org]. Question 1. CRPP has a national mandate as the executive branch of the Association Euratom-Swiss Confederation. Our theory group has been involved in HPC for decades and is now one of the active partners in the Blue Gene Project at EPFL. The most demanding computing needs of CRPP are (and will be) in the areas of first-principle based simulations of turbulence and of kinetic effects on fluid instabilities in magnetic confinement systems. When ion turbulence alone is considered, we estimate the need for one full-scale simulation of an ITER-size plasma to be equivalent to 4 BG/L racks (8192 processors) for one week. Extension to include the physics related to electron dynamics is being performed. This extrapolation yields at least an order of magnitude higher requirement. One should stress that all our high-end applications are homegrown codes, developed in collaboration with Research Institutes in EU, USA and Japan. It is therefore important that the availability of a many hundreds teraflops machine is insured in the next few years. The issue of the timely availability of a Petaflop/s system must be an element in the final decision. Question 2. The role of the CSCS should be to provide HPCN services to the widest spectrum of scientific applications. Scientific and technical competence in HPCN should remain distributed and be strengthened in the various Research Institutes throughout Switzerland. The CRPP has achieved its leading role in July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-67 ETH Board CENTRE DE RECHERCHES EN PHYSIQUE DES PLASMAS page 4 the fusion community by applying this strategy. CSCS or any other “Centres” cannot and should not be used to centralize HPCN competence. Past experiences have confirmed our views. Question 3. The main technical issue is that of power efficiency. [Example ORNL: “ petascale system could require an additional 10MW by 2008 just for computers add 75% for cooling]. A relevant figure of merit is the System Power Efficiency (Flop/s per Watt for real applications) Question 4. The most effective measure to enhance academic contributions to HPCN, or rather to enhance HPCN contribution to academic output, is to have a longterm commitment at the highest political level for the development of HPC in Switzerland. This commitment should not be restricted to support of the CSCS but be supportive of the knowledge base in (and by) the academic institutions in general and in laboratories active in the field in particular. The Petaflop/s system should not be a “one-off” initiative. It will be successful if and only if the education and build-up of competencies, notably in parallel programming, is pursued in the academic institutions without disruption. The time scale for this build-up is much longer than the lifetime of a given platform and the time scale of the evolution of top-end computer architectures. This implies that funding for hardware should be accompanied by long-term commitment to support scientific staff to develop, and maintain the competencies. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-68 ETH Board Statement regarding Swiss National Strategic Plan on HPCN Switzerland has always been a pioneer in computational sciences, starting half a century ago with the purchase of the Zuse Z4 by Stiefel, and has consolidated its leadership role in the field in the following decades – a role that has been quantified by being the country with by far the largest supercomputing power per capita, and demonstrated by the pioneering curriculum in computational sciences and engineering developed at ETHZ. In the last decade Switzerland has gradually lost that role. At least one worldrenowned computational scientist has left Switzerland because of a decline in computational resources. Since the economy and progress of Switzerland relies on the innovative power instead of natural resources, it would be disastrous if Switzerland were to give up its role as one of the top players in the field of computational sciences, just as the field is maturing and growing in importance. Given the fact that Switzerland has the resources to remain a leader in the field, we cannot afford not to invest in this field to keep our position. In areas such as weather and climate prediction, materials simulation and design, device simulations, and fluid dynamics, the importance of supercomputer simulations is steadily growing. At the same time new roles for computer simulations emerge in the area of risk assessment. All of these fields are of prime importance to the Swiss financial and manufacturing industry, with a large growth expected in the area of risk prediction and minimization. Some efforts have direct practical importance while others are long-term basic research efforts laying the foundation for future technological breakthroughs. It is therefore important that Switzerland acquires a high-end supercomputing system with world-wide recognition, to regain its role as a serious partner. Petaflop-level will certainly be the order of magnitude needed to stay competitive by 2011. However, even more important than hardware is the investment in new algorithms, new libraries, software development and maintenance. Significant scientific breakthroughs will be possible already on 100 Gflop machines, if sufficient funding for algorithmic and software development and maintenance were made available. Future machines with more than 100’000 nodes will require even more effort to be invested into algorithmic and software development, both of general-purpose libraries and of application-domain specific codes. Currently there is no source of funding for such activities in Switzerland, and development has to be cross-subsidized from research grants, which is not ideal since this prevents focusing on high-quality sustainable development and instead favors quick scientific returns with less-than-optimal codes. A widespread mistake to avoid is investing huge resources into hardware, and only tiny fractions of into manpower. In my own research field, the growth of algorithmic performance over the past twenty years greatly exceeds that of the exponential growth of raw hardware compute power. 1980’s computers with current algorithms would be faster than current computers with 1980’s methods! To use the words of a colleague, we need not only supercomputers but also “super-libraries”, “super-tools” and “superapplications”. Given a fixed amount of money, the plan with the best return would be to spend 50% on hardware and 50% on software development/manpower, instead of a 99% - 1% division that is a common mistake. Furthermore, while the algorithmic and software achievements have lasting impact, the hardware is outdated after 5 years. Significant funding for algorithmic and software development will also enhance the academic contributions to HPCN, and will help to establish Switzerland as a leader in supercomputing – we cannot compete in building the fastest computer but we can excel at producing the best supercomputing software and tools. The optimal role of CSCS will be to focus on high-end computing hardware that cannot be afforded by individual institutions, and to provide computational and methodological support to Swiss research teams. Prof. Matthias Troyer, ETH Zürich July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-69 ETH Board Ten thousands of CPUs Tens to hundreds of CPUs Level of tightly coupled parallelism (as of 2006) July 4, 2007 National Supercomputing Centre CSCS High-Capability Supercomputers High-Capacity Cluster Computers Research Labs Universities SNSP-HPCN | Final Report | Appendix G Higher Education Centres single CPU page G-70 Ulmer & Sawley ETH Board •The question is not whether but when Switzerland needs a petaflop system. Needs ramping up for people, SW and infrastructure; feasible date: 2011? July 4, 2007 •For maximising the capability of HPC resources in CH, concentration must take place. National leading house must exist as international representative. This leading house which could host special purpose systems for single institutions SNSP-HPCN | Final Report | Appendix G •Driving HPC needs experimentation with upcoming technology (which is part of CSCS’ mandate) -> CSCS Strategy 2007-2015 •Switzerland needs a curriculum in Scientific High-Performance Computing; CSCS must contribute to this curriculum •We don’t need flashy single-shot projects, but a sustainable national HPC programme page G-71 HPCN Open Day, Bern -, Dominik Ulmer, 6. December 2006 ETH Board PAUL SCHERRER INSTITUT Villigen, December 4th 2006 Helena Van Swygenhoven Bldg. WBBA/114 CH-5232 Villigen PSI Switzerland Phone Telefax E-mail: operat. 056 / 310 21 11 direct 056 / 310 29 31 central 056 / 310 21 99 direct 056 / 310 31 31 helena.vs@psi.ch To: Prof. R. Eichler Cc: K. Baltensberger Response on the SNSP-HPCN questionnaire 1. Switzerland should have as goal to acquire a peta-flop system, however because the size of research budgets and the technological demand, acquisition should not be the immediate goal (2008-2011). The knowledge base at international level for effective use of petaflop platforms needs to be developed to avoid the use of peta-flop computers as a simple ensemble of tera-flop computers. Switzerland has high intellectual resources that should play an important role in the developments in the field of I/O, data mining, data visualization etc. necessary for the effective use of peta-flop computers. In other words, we should not jump blindly in the peta-flop race…. 2. A peta-flop (but in principle in general valid for a HPCN)) computer center should be at national level, functioning as a userlab. The role plaid by CSCS now is not optimal, certainly not if one thinks in the direction of peta-flop computers. CSCS should concentrate on providing users support for running their programs on the platforms, on hardware and fine tuning of the machines and not on software such as visualization tools etc. Application performance should be put in a “industry (IBM,Cray..)/computer center/user ” interface philosophy. 3. Currently Switzerland is not at the level for operating a peta-flop system in terms hardware (for instance power). For data transfer, storage and analysis a lot has to be done for effective use of peta-flop computers, but this statement is also valid at international level. In my opinion this is one of the main reasons for not considering the acquisition as an immediate goal (which would risk a dump of all research money), but rather to concentrate on the necessary skills for usage. 4. Incorporation at the national level within a research environment but also through specific academic appointments, guaranteeing our intellectual competitiveness through appropriate teaching. July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-72 ETH Board -2- 5. New HPC facilities should be integrated into a national center as suggested in 4. Although the existence of internet facilities, grids etc, human presence can not be replaced. The lack of a scientific and/or academic environment influences negatively the performance of CSCS. The operation of a HPC center is highly demanding on technical as well on academic level and the operation as a userlab with close contact with research/academic environment is a must to guarantee meaningful applications. 6. The center should be at national level and financed at national level, which .makes a credit system unnecessary. 7. Industry should have the opportunity to participate as well in development as in usage. 8. no opinion Prof. Dr. Helena Van Swygenhoven Paul Scherrer Institute Prof. titulaire EPFL Switzerland July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-73 ETH Board Rückmeldung Rolf Wohlgemut / Vizedirektor Siemens Schweiz Sehr geehrter Hr. Baltensberger Hr. Fiorenzo Scaroni hat mich angefragt, betreffend der nationalen HPC- Strategie unsere Stellungnahme abzugeben. Gerne möchte ich Ihnen wie folgt antworten. Die Siemens Schweiz AG hat in der Vergangenheit noch keinen Bedarf an einer zentralen Supercomputing Leistung gehabt. Auch sehen wir für die nähere Zukunft ebenfalls keinen Bedarf. Wir könnten uns als globaler Konzern eher eine interne Gridlösung bei Bedarf vorstellen, um unsere weltweit verteilen Computerleistungen zu bündeln. Auf der anderen Seite begrüsst Siemens, dass in der Schweiz eine Gesamtstrategie im Supercomputing entwickelt wird. Diese soll für die Schweiz hohen Investitionen und den Betrieb von teueren Ressourcen optimieren. Eine wichtige Voraussetzung ist zudem, dass diese Supercomputing Ressourcen in das akademische Europäische Netzwerk Géant2, wo die Schweiz prominent vertreten ist, eingebettet werden. Ebenfalls möchte ich mich für die Einladung vom 6.Dez. bestens bedanken. Leider muss ich Ihnen mitteilen, dass ich am 6.Dez. nicht kommen kann. Freundliche Grüsse R.Wohlgemuth ******************************************************** Siemens Schweiz AG Corporate Technology I-41 / CT Albisriederstr. 245 CH-8047 Zürich Dr. Rolf Wohlgemuth Vizedirektor Email: Voice: Fax: Internet: rolf.wohlgemuth@siemens.com +41 585 586 560 +41 585 585 986 http://www.siemens.ch July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-74 ETH Board July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-75 ETH Board July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-76 ETH Board Input from Mihaela Zavolan, Uni Basel Hi Manuel I read the materials that you sent, but I am not sure how I can contribute. The questionaire covers mostly strategic questions, and I am not sufficiently informed about the landscape of computational needs and resources in Switzerland to be able to comment about such things. Concerning the more specific points, about which I can comment, please see below. In general, I think it is essential to have an evolvable computational infrastructure. I also think that it is essential to organize it efficiently, given that the resources are always limiting. This concerns the machines as well as the personnel operating them. Whether Switzerland should acquire a peta-flop system.. in the long run it will happen. Whether it should be now or later, that depends on the user demands in the foreseeable future. As for the role of CSCS, that depends on too many things I do not know. Technical issues: you have basically mentioned them. As the system will obviously have to serve the whole of Swizerland, key points will be sufficient (and easily expandable) storage, fast search and retrieval of large amounts of data from a large storage system, all the issues associated with backing up such a system, data protection etc. To some extent, the solutions depend on the position of CSCS with respect to, for example VitalIT and the computing infrastructure of SystemsX. To enhance the academic contributions to HPCN in Switzerland it is clearly required that this area is well covered in the curricula of the computer science departments. I think it will also be a good idea to have (maybe there are already?) internships within CSCS. This may of course raise issues concerning security of the system, and the data, which will need to be addressed. Link between CSCS and the scientific community: that will probably only be built/kept based on collaborations, so the scientific community needs to learn what services they could use from CSCS. I do think that the universities should have credits in HPCN (aren't there already?). As for national services.. From my perspective, one could think of the model of NCBI, and DDBJ. But this is probably too specialized, already covered to a great extent by SIB, and probably overlapping with EBI. That's, I think, all I can say now. Unfortunately I will not be able to attend the meeting on Wednesday. Thanks for your email and best Mihaela July 4, 2007 SNSP-HPCN | Final Report | Appendix G page G-77