Title: A multicore-enabled framework for communication and I/O in next generation clusters
Keywords: Multicore, I/O, communication, MPI, file system.
Parallel architectures are becoming increasingly complex and hierarchical. While many recent efforts in this field focus on designing programming environments and libraries for large scale configurations (Clusters and Grids), the most impressive hardware evolution is currently taking place inside cluster nodes themselves, with the emergence of multicore chips. By a multicore chip, we mean that many processor cores (or cores for short) are integrated into one chip. Today's high-end clusters usually use 16-core nodes and feature more than 10,000 cores. It is expected that the amount of cores per node will dramatically increase in the upcoming years, which will deeply modify the structure of clusters: they will move from large clusters of PCs to smaller clusters of powerful multicore machines. However, the other hardware components did not evolve so swiftly. In particular, each cluster node is only equipped with a small shared pool of network interface cards. As a consequence, remote IO operations and communication operations have to face a major bottleneck problem. Since this issue is inherent to the hardware, it actually affects most existing programming environments such as MPI libraries. As a result, the gap between theoretical peek and sustained performance of clusters is still widening.
Significant progress has recently been achieved about improving communication within clusters of multicore nodes. One approach, designed by Prof. Namyst's group, is based on a low-level flexible communication scheduler that can apply various packet-level optimizations. This work has demonstrated that careful optimizations at the runtime-level can dramatically increase network utilization and minimize the bottleneck problem that typically arises when multiple processes simultaneously access a small pool of shared network interface cards. Moreover, new results regarding efficient IO mechanisms for clusters of multicores have recently been achieved by Prof. Ishikawa's group. In a cluster of 10,000 multicores, for example, 10,000 remote file IO requests may be independently sent to the single file server via network. Such a large amount of independent IO requests causes low utilization of both the network and file server due to congestion. In order to avoid such low utilization, Prof. Ishikawa's group has developed a new mechanism to coordinate IO requests in order to reduce the network trafic and the file server load.
By comparing our respective approaches, we have realized that most of our optimization techniques are very similar in the sense that requests are gathered and arranged based on some application-specific optimization policy. The challenge is now to design a unified multicore-enabled framework to integrate and optimize both communication and remote file IO. Such a framework will form a great research vehicle to develop new optimization policies for the future multicore technologies. This research proposal has two main goals: providing a unified framework for dealing with IO and communication together with a common database of optimization strategies, and addressing a large spectrum of programming paradigms, such as multithreading, pure message-passing or hybrid programming. The first research topic aims at providing general abstractions that can model the behavior of communication and remote IO. By doing this, optimization algorithms can be defined and applied to IO flows, communication flows, or both (e.g. aggregating IO data and communication). The second research topic is to build this multicore-aware software so that it can benefit to a wide range of programming environments, including pure MPI implementations and hybrid ones. In other words, we aim at providing efficient and scalable coordination of IO and communication on multicore architectures through a generic library. We eventually intend to integrate this library into existing, widespread high level programming environments including MPICH2 and PVFS2 (for MPI-IO).
Although the whole project will probably need more than two years to be completely finalized, most results will be obtained during the first two years. Hence a research timetable that spawns over two years. The projet will feature 4 main workpackages:
The overall aim is to join our efforts in order to extend the scope of some important research results we achieved recently, and to provide a software framework to provide programmers of higher level programming environments with a highly versatile, multicore-aware runtime system. The research skills of the two groups somehow overlap regarding communication over high speed networks, but the methods they use and the application domains they target are trully complementary. Moreover, this project is based on merging and generalizing the ideas of Pr. Ishikawa about the optimization of remote I/O and the ideas of Pr. Namyst about the optimization of communication:
Professor Ishikawa's group has been designing and implementing an efficient remote I/O facility called PGAS. This group also has been designed and implemented from scratch two types of the MPI communication library. The first one is called YAMPI while the other one called GridMPI is used in a Grid environment. Prof. Ishikawa's group has also been developing a communication model to express the communication behaviour under the TCP/IP protocol.
Professor Namyst's group has a long standing research activity in the field of communication over high speed networks. The group has a significant experience in designing optimized MPI implementations and low-level communication libraries. The group also have much experience in scheduling threads over multicore architectures. Moreover, Prof. Namyst's group did establish that using multithreading within communication and progress engines can significantly improve the overall performance by allowing the implementation of opportunistic, just-in-time optimizations. The design of the NewMadeleine communication library stems from this idea and features a flexible network packet scheduler that can rearrange packets in a arbitrary manner.
Both sides now have significant insights about how both approaches could be unified, and how higher-level environments (typically MPI) could benefit from such a software framework. The necessary collaboration between our groups will heavily rely on frequent mid-term visits of PhD students from both groups.
Benefits are twofold. First, this collaboration will allow french researchers to acquire state-of-art knowledge in the domain of remote I/O subsystems and it will also be a great opportunity for PhD students to broaden their scope and generalize their ideas to new application fields. Second, our research team develops low level runtime systems for parallel architectures, mostly focusing on performance rather than easy to use programming interfaces. Thus, the success of such runtimes greatly depend on their successful integration into higher-level, standard programming environments (such as MPI implementation for communication, or parallel file systems for I/O). This project will offer a good opportunity to do so, in collaboration with japanese partners that have a high expertise in this domain.
Based on the collaboration, YAMPI implementation and remote I/O facility, Prof. Ishikawa’s group software products, will be extended and evaluated using the new framework. Those products will be integrated into the SCore cluster system software which has been used in the world wide, and become de facto standards in the next generation parallel architectures. The SCore has been developed at the PC cluster consortium headed by Prof. Ishikawa.
Yutaka Ishikawa, Professor at University of Tokyo
Hiroya Matsuba, Research Associate at University of Tokyo
Akihiro Nomura, Ph.D candidate student at University of Tokyo
Kazuki Ohta, Master student at University of Tokyo (will be Ph.D candidate student in 2010)
Tomoya Adachi, Master student at University of Tokyo (will be Ph.D candidate student in 2010)
Raymond NAMYST, Professor at University of Bordeaux 1
Alexandre DENIS, Research Scientist at INRIA
Guillaume MERCIER, Lecturer at University of Bordeaux 1
Francois TRAHAY, Ph.D candidate student at University of Bordeaux 1
Jerome CLET-ORTEGA, Ph.D candidate student at University of Bordeaux 1
In the context of the French national Grid-5000 initiative, four high performance clusters have so far been installed in Bordeaux, featuring a total of more than 600 cores. One of these clusters features an interesting architecture composed of 10 8-core nodes connected by a heterogeneous multirail configuration (Myri10G + Infiniband + GigaEthernet). In addition to these computing resources, a hiearchical multicore machine shall be installed in december 2008 at the INRIA Bordeaux research center and dedicated to Raymond Namyst's group research experiments
Professor Ishikawa is partially involved with the Information Technology Center at the University of Tokyo, where a 140 TFlops cluster system is operationed. Thuis system features 956 nodes, each of which consists of 16-cores CPU, connected by either four or two Myri-10G NICs. The French group may also use this machine via the Internet.
[1] François Trahay, Elisabeth Brunet, Alexandre Denis, and Raymond Namyst. A multithreaded communication engine for multicore architectures. In CAC 2008: Workshop on Communication Architecture for Clusters, held in conjunction with IPDPS 2008, Miami, FL, April 2008. IEEE. To appear.
[2] François Trahay, Alexandre Denis, Olivier Aumage, and Raymond Namyst. Improving reactivity and communication overlap in mpi using a generic i/o manager. In Capello, Herault, and Dongarra, editors, EuroPVM/MPI, volume Recent Advances in Parallel Virtual Machine and Message Passing Interface of Lecture Notes in Computer Science, pages 170 - 177. Springer, 2007.
[3] Olivier Aumage, Elisabeth Brunet, Guillaume Mercier, and Raymond Namyst. High performance multi-rail support with the newmadeleine communication library. In HCW 2007: the Sixteenth International Heterogeneity in Computing Workshop, held in conjunction with IPDPS 2007, Long Beach, California, USA, March 2007.
[4] Elisabeth Brunet, Olivier Aumage, and Raymond Namyst. Dynamic optimization of communications over high speed networks. The 15th IEEE International Symposium on High Performance Distributed Computing, Paris, June 2006.
[5] Brice Goglin. Design and Implementation of Open-MX: High-Performance Message Passing over generic Ethernet hardware. In CAC 2008: Workshop on Communication Architecture for Clusters, held in conjunction with IPDPS 2008, Miami, FL, April 2008. IEEE. To appear.
[1] Kazuki Ohta, Hiroya Matsuba, and Yutaka Ishikawa, "Gather-Arrange-Scatter: Node-Level Request Ordering for Parallel File Systems on Multi-Core Clusters", IEEE International Conference on Cluster Computing, 2008 (poster).
[2] Akihiro Nomura, Hiroya Matsuba, and Yutaka Ishikawa, "A Network Performance Model for TCP/IP-based Cluster Computing" , IEEE International Conference on Cluster Computing, 2007.
[3] Motohiko Matsuda, Tomohiro Kudoh, Yuetsu Kodama, Ryousei Takano, and Yutaka Ishikawa, "Efficient MPI Collective Operations for Clusters in Long-and-Fast Networks" , IEEE International Conference on Cluster Computing, 2006. (Best Paper)
[4] Motohiko Matsuda, Tomohiro Kudoh, Yuetsu Kodama, Ryousei Takano, and Yutaka Ishikawa, ``TCP Adaptation for MPI on Long-and-Fat Networks'' , IEEE International Conference on Cluster Computing, 2005
[5] Motohiko Matsuda, Yutaka Ishikawa, and Tomohiro Kudoh, ``Evaluation of MPI Implementations on Grid-connected Clusters using an Emulated WAN Environment'', CCGRID2003, 2003
During the course of this project, we intend to apply for Postdoctoral fellowships both on the Japanese side (JSPS Postdoctoral fellowship program) and on the French side (INRIA postdoctoral fellowship program).
Both research groups (i.e. groups lead by Prof. Ishikawa and Prof. Namyst) have a long standing research activity in the field of parallel runtime systems and communication libraries for high speed networks. Thanks to the JSP-CNRS NEGST initiative (NExt Grid Systems and Techniques), a series of franco-japanese workshops aiming at promoting collaborations between Japan and France, numerous exchanges between groups have occured since 2006. At this time, Prof. Ishikawa was working on his YAMPI MPI implementation for high speed networks, and Prof. Namyst was designing the PadicoTM middleware for communication over heterogeneous hardware configurations (e.g. Cluster of clusters or Grids). During a visit to Japan by two members of Prof. Namyst's Group (April 2007), we have developed a prototype integrating PadicoTM and YAMPI, allowing YAMPI to run seamlessly over both high-performance multi-rail clusters and grid networks (including routing, firewall crossing).
To set up the basis of a more ambitious collaboration between both sides, Prof. Raymond Namyst spent two months at the University of Tokyo (from August to September 2008) to work with Prof. Yutaka Ishikawa. This visit was supported by the JSPS invitation fellowship program for research. During the stay, the numerous research meetings we had gave us many insights about how our complementary research works could integrate and lead to an ambitious new research project. We first started some experimental work to evaluate the gain we could achieve by re-ordering remote I/O requests over a high speed network. The positive results we have obtained and the similarity between this problem and the communication optimization one lead us to imagine a unified framework to tackle both challenges by using a common optimization engine.
This project proposal aims at designing, implementing and evaluating the aforementionned runtime framework.
François Trahay is a (3rd year) PhD candidate within the Runtime group. He has been working on improving reactivity to communication events on multicore cluster nodes. At first, he has designed a low-level scalable I/O event manager that asynchronously detects the completion of communication requests thanks to a tight integration with the thread scheduler. Then, he has designed a event-driven architecture for multicore machines that enables higher level environments to easily request the execution of threads or tasklets over specific subsets of cores, with various priorities. He successfuly demonstrated [CAC2008] that his runtime system could be used by communication libraries to opportunistically offload independent, time-consuming communication processing over available cores. This project proposal clearly opens new perspectives for the work of François. Not only will he be involved in the collaboration during the last year of his PhD, but he will also apply to postdoctoral position at University of Tokyo during the second year of the project (April 2010).
Jérôme Clet-Ortega is a (2ng year) PhD candidate that works on a programming environment that allows the flexible execution of hybrid applications mixing MPI and threads (OpenMP, TBB or Pthreads). Applications can arbitrarily choose the granularity of parallelism by using any configuration going from one MPI process per machine to one MPI process per core. Obviously, performance of the runtime implementation heavily relies on its ability to coordinate communication and I/O requests within each node. Jérôme will thus be greatly involved in the "integration to high level middleware" and "evaluation" phases of this project. He should spend several weeks at University of Tokyo during the course of the project.
The following people will design and implement the proposed software and write papers with Prof. Ishikawa:
We obviously intend to publish our research achievements in several major conferences in the field of high performance computing (IEEE International Conference on Cluster Computing, IEEE International Parallel and Distributed Processing Symposium, IEEE/ACM Supercomputer Conference). More precisely, if the project makes progress as expected :
The software developed at this project will be distributed to the world-wide community so that a significant number of researchers and developers could give the proposed framework a try. On the Japan side, the software will be integrated into the SCore cluster system software developed at the PC cluser consortium whose chairman is Prof. Ishikawa.
Nothing planned yet.
Our research groups are actually involved in the most significant software developement projects related to the Message Passing Interface.
The french side has a long standing collaboration with the Radix Lab of Argonne National Laboratory (IL, USA) about the efficient support of high speed networks within the MPICH implementation. Since 2006, the runtime group is actively involved in the development of the MPICH2-Nemesis runtime system. This collaboration between the Radix Lab/Argonne and the Runtime group has been officially supported by INRIA since January 2007. Currently, they are focusing on enhancing the support for multithreading within MPICH2 to take advantage of multicore architectures.
In the Japanese side, Prof. Ishikawa's group has been developing the YAMPI MPI communication implementation which uses the Center for Computational Sciences at the University of Tsukuba. They are also currently developing an interoperable network protocol for MPI and integrated it to one of the de facto standard MPI implementations, called OpenMPI which has been implemented by universities, research institutes, and companies, such as Indiana University, University of Stuttgart, Los Alamos, Sandia National Laboratory, Oak Ridge National Laboratory, INRIA, SUN, and IBM. Prof. Ishikawa is a member of the MPI standard committee where the features of MPI-2.1, MPI-2.2, and MPI-3.0 have been discussed for standardization.
As a result, there clearly appears to be a lot of common research interests between the Japanese, the French and the American groups. So we hope that the outcomes of this project will contribute to providing the MPI community with a common, scalable software framework that would integrate optimizations for multicore architectures and would ease the development of new optimization policies.
The new framework, communication library, and remote I/O mechanism developed in this project will be integrated into the SCore cluster system software developed at the PC cluster consortium whose members are major Japanese computer vendors including Fujitsu, Hitachi, and NEC.
Prof. Ishikawa is the chairman of the PC cluster consortium. The SCore cluster system software has been distributed to the world-wide communities. A company in England has a business using SCore. In Japan, SCore has been used at two supercomputer centers, the University of Tsukuba and RIKEN.
The Runtime group has strong collaboration links with CEA (French Atomic Energy Commission) in developing joint software for large, multicore clusters. The runtime system we are intending to develop in the course of this project will undoubtly be evaluated by the CEA and, if the experiences are successful, will probably be integrated in their software suite.
| Name | Purpose | Number of Trips | Days | Destination |
|---|---|---|---|---|
| Raymond Namyst | Overall synchronization | 1 | 7 days | Japan |
| Guillaume Mercier | WP3 coordination | 1 | 7 days | Japan |
| Alexandre Denis | WP1 coordination | 1 | 7 days | Japan |
| François Trahay | Joint development | 2 | 14 days | Japan |
| Jérôme Clet-Ortega | Joint development | 1 | 7 days | Japan |
| Name | Purpose | Number of Trips | Days | Destination |
|---|---|---|---|---|
| Yutaka Ishikawa | 1 | 7 days | France | |
| Hiroya Matsuba | 1 | 7 days | France | |
| Akihiro Nomura | 1 | 7 days | France | |
| Kazuki Ohta | 2 | 14 days | France | |
| Tomoya Adachi | 1 | 7 days | France |
Current position
Professor at Université Bordeaux 1 (France) since September 2002.
Research Interests
Raymond Namyst is the scientific leader of the Runtime INRIA group, devoted to the design of high performance runtime systems for parallel architectures. His main research interests are in parallel computing, thread scheduling algorithms and parallel languages for multiprocessor architectures (SMT, multi-cores, NUMA), communications libraries for high speed networks (Myrinet, Infiniband, Quadrics, SCI) and communications within Grids (Raymond Namyst was the local scientific coordinator of the Grid5000 project at University of Bordeaux from 2003 to 2007). He has played a major role in the development of many software (Marcel, Madeleine, PadicoTM, MPICH/Mad, etc.) that form the PM2 software suite.
Raymond Namyst has supervised 13 Ph.D thesis, and has has written numerous papers about the design of efficient runtime systems (please visit his website for more information).
Professional Background
Social Activity
Education
Research interests
Raymond Namyst is the scientific leader of the Runtime INRIA group, devoted to the design of high performance runtime systems for parallel architectures. His main research interests are in parallel computing, thread scheduling algorithms and parallel languages for multiprocessor architectures (SMT, multi-cores, NUMA), communications libraries for high speed networks (Myrinet, Infiniband, Quadrics, SCI) and communications within Grids (Raymond Namyst was the local scientific coordinator of the Grid5000 project at University of Bordeaux from 2003 to 2007). He has played a major role in the development of many software (Marcel, Madeleine, PadicoTM, MPICH/Mad, etc.) that form the PM2 software suite.
Raymond Namyst has supervised 13 Ph.D thesises, and has written numerous papers about the design of efficient runtime systems (please visit his website for more information).
Current position
Professor at the University of Tokyo Head of computer sicence department
Professional Background
Social Activity
Education
Research Experience
Prof. Ishikawa has been focusing on realization of a multi operating system kernel environment for multi-core processors so that different kernels such as regular Linux and real-time system kernels run on a single machine. Unlike the virtual machine (VM in short) approach, CPU and memory resources are partitioned by multiple native kernels without any architectural virtualization supports. There is nearly no slowdown unlike VMs, because the kernel and user programs are executed directly by the real CPUs. His student has implemented the prototype system that the two independent Linux kernels run on a quad-core processor.
Prof. Ishikawa has been involved with the development of GridMPI which is an MPI-2 implementation for the Grid environment. GridMPI is implemented on top of the YAMPI MPI implementation which Prof. Ishikawa has been developing. In the Grid environment, both network latency and bandwidth have to be taken into account to provide the high performance communication environment. One of the contributions of this research is that we have revealed the existing TCP protocol handling problems for MPI communication traffics over the Grid environment, and proposed a new implementation. In order to realize interoperability among MPI implementations, GridMPI supports an extended IMPI (interoperable MPI) protocol. We are currently porting the extended IMPI protocol to the OpenMPI implementation (http://www.open-mpi.org) to contribute to the Open MPI activity.
Since 1993, Prof. Ishikawa has been responsible for designing and developing the SCore Cluster System Software that is a high-performance parallel programming environment for Linux PC clusters. The SCore development was started at the Real World Computing Partnership (RWCP) funded by the Ministry of Economy, Trade and Industry (METI) of Japan. After the end of the RWCP, the PC Cluster Consortium, supported by major Japanese computer vendors and integrators, took over the development of SCore (http://www.pccluster.org/). Prof. Ishikawa is the chairman of the consortium.