Speech | How To Build A Complete High-Performance Computing Cluster Architecture?
2022-05-26

 

 

Introduction

A complete High-Performance Computing cluster requires a high-capacity and highly scalable storage system. Storage is of paramount importance in cluster construction.

ClusterTech’s Senior HPC Solution Engineer, Ma Xiaohui, was invited by Seagate to share online on 'How to Build a Complete High-Performance Computing Cluster Architecture', and to interpret the unique advantages of the joint solutions between ClusterTech and Seagate in HPC cluster construction.

We have compiled the speech transcript of this live broadcast for everyone's reference and learning. The following is the original text for sharing:

“Hello everyone, I am Ma Xiaohui, an engineer from ClusterTech. I am very pleased to be able to share some of ClusterTech's experiences and insights in the field of igh-performance computing with you all.”

Here is my presentation.

 

What can High-Performance Computing do?

Here are some applications of High-Performance Computing.

For a typical application like a weather forecasting system, it is impossible for a meteorological agency to use just one or two computers to compute weather data. Even if the forecast is accurate, it will be meaningless. It is necessary to predict the weather for the next two hours within 10 or 20 minutes, and that is what makes it meaningful.

This includes tasks such as pneumatic analysis for manufacturing enterprise users and car design for automotive manufacturing companies. If a small amount of computing power is used, it may take several days to obtain the corresponding results. However, by using a cluster, the same result can be obtained in 1-2 days.

Similarly, in the field of bioinformatics, including the current COVID-19 pandemic, machine learning and GPU-accelerated heterogeneous computing are used in areas such as genetic testing analysis and protein prediction. These applications are based on high-performance computing architectures.

In addition, High-Performance Computing is also used in traditional scientific research fields such as molecular dynamics simulations in the field of computational chemistry and material simulations in materials science. These applications require high computational power to accurately model and simulate complex systems and phenomena.

So, High-Performance Computing clusters address the problem of how to quickly and accurately obtain large-scale computational results.

 

What business does ClusterTech do in the field of High-Performance Computing?

Briefly speaking, ClusterTech has been focusing on High-Performance Computing for over 20 years. With a focus on applications and deep integration with user business logic, ClusterTech provides solutions from the system layer to the application layer to help users address the challenges of buying and using clusters effectively.

ClusterTech provides users with a complete solution for building High-Performance Computing clusters, including the setup of hardware and application software, platform optimisation services, and management software for the HPC cluster platform. In other words, they provide users with a comprehensive solution that combines hardware and software for building HPC clusters.

A complete cluster not only requires a powerful hardware configuration system, such as CPUs, memory, disks, and networks, but also a comprehensive software platform to integrate, support, monitor, and manage the resources of the cluster. Users then run their corresponding application software on this cluster to complete calculations and achieve the ultimate goal of High-Performance Computing.

High-Performance Computing clusters are a relatively specialised and niche field, and in addition to developing and implementing HPC scheduling software, ClusterTech is responsible for delivering complete HPC platform solutions. This includes everything from deploying the operating system to managing the cluster platform, optimising user application software, and providing overall cluster operations and maintenance, offering users a "one-stop-shop" HPC service.

 

How to truly establish a High-Performance Computing cluster?

In terms of networking, High-Performance Computing clusters require the highest performance. In HPC architecture, the network is typically divided into management, storage, and compute networks. In HPC, the network often uses InfiniBand or high-speed Ethernet networks instead of 1G or 10G Ethernet, as the bandwidth and latency of these latter options may not be sufficient to meet the requirements of High-Performance Computing.

In High-Performance Computing, computational resources are managed through a cluster that includes management servers, login servers, and computational servers. The computational servers include CPU-based servers, GPU-based heterogeneous computing servers, graphical and visual processing nodes, and "fat" nodes, which are nodes with large memory, four-way or eight-way CPU cores, and other configurations. Different types of nodes are configured according to the specific application requirements of the users.

High-Performance Computing clusters typically use a distributed storage system architecture. The storage solutions provided by Seagate have significant advantages in meeting the requirements of HPC clusters, such as high performance, stability, data security, and cost-effectiveness.

In the entire HPC architecture, the performance of application software is the most important and highly valued aspect for users. All software and hardware components are designed to serve the applications, and the entire cluster is aimed at improving application efficiency.

Therefore, the characteristics of the applications used by users in High-Performance Computing clusters determine the selection of the cluster system. For example:

If the application is CPU-intensive, the choice between high-frequency cores with fewer cores or lower-frequency cores with more cores needs to be considered. For CPUs, the single-core frequency and the number of cores in a single CPU are differentiated based on factors such as different levels, generations, and TDP power consumption. Therefore, different types of users’ applications have specific requirements for CPU selection.

At the network level, high-bandwidth or low-latency networks are generally chosen to meet the specific application requirements of users.

In terms of storage, as mentioned earlier, different storage configurations are recommended for different types of storage to meet the usage scenarios of applications involving large file sequences or massive small files. Memory considerations must also be taken into account within the entire High-Performance Computing cluster.

High-Performance Computing cluster users have a specific budget, and therefore, it is necessary to have appropriate combinations of CPU, memory, network, and storage based on the characteristics of their applications.

Different application types of users may result in slight differences in the CPU selection. For example:

If a user's application is single-threaded or has low efficiency in multi-core parallelism, it is more suitable to choose a high-frequency CPU with fewer cores to meet the task-level parallelism requirements. Storage and network layers also need to be properly planned within the given budget; otherwise, situations like the following may arise.

For instance, if a user's task requires significant I/O but the storage aspect is not given enough attention, and more focus is placed on computational power, resulting in a well-equipped CPU. However, when a large number of tasks are submitted, it may lead to an I/O storm, causing the CPU to be underutilised and overall task efficiency in the cluster to decrease. In such cases, it may be discovered that using 6 machines with a good storage system performs significantly better and more efficiently than using 12 machines with a poor storage system.

Therefore, when it comes to the overall architecture of an HPC system, there are many factors to consider in order to build a highly functional cluster.

Regarding node types, the High-Performance Computing solution offered by ClusterTech supports CPUs from Intel, AMD, as well as ARM-based CPUs using the domestic MIX architecture. Furthermore, the solution has been adapted to be compatible with various domestic and international operating systems.

Additionally, ClusterTech's cluster software management platform enables unified management of all devices, ensuring the availability of cluster resources and future scalability.

In terms of network types, it is generally recommended that users utilize a minimum of 25GB Ethernet networks to support their High-Performance Computing clusters. This is because, from the current stage or based on experience, 10GB Ethernet networks are often insufficient to handle the computational demands.

However, there are still some clusters that were built a few years ago using 10GB Ethernet connections. In practice, when running large-scale grid computing or high-precision calculations, the amount of data being transferred between nodes is significant, and the bandwidth can become a bottleneck, leading to underutilisation of the CPU.

Storage is indeed a critical component in a cluster. In addition to discussing CPUs and networks, let's now talk about storage.

Regarding our High-Performance Computing cluster platform and Seagate Technology's hardware platform, compatibility and adaptability tests have been conducted. In terms of storage, we can choose cost-effective or high-density models to meet our cluster storage needs:

For small-scale clusters, the Seagate 2U12 or 2U14 storage platforms can be used, providing storage capacities of several hundred terabytes (TB). This generally satisfies the read/write performance requirements of NFS or distributed storage for around 10 to 20 nodes.

For medium-scale clusters, two or four 5U84 Seagate storage platforms can be used to implement distributed file system data segment storage. The 2U12 platform can also meet the primary data storage needs.

For massive high-density storage requirements, intermediate densities ranging from several petabytes (PB) to several hundred terabytes (TB) can be satisfied. For higher-density scenarios, the 4U106 platform is recommended to meet the demands of ultra-large-capacity distributed file systems. As a single 4U106 unit can provide approximately 1 PB of usable capacity, similar to the 5U84 platform, using two or four units can reach storage requirements of 3 PB or 4 PB, respectively, for massive storage needs.

Storage software is indeed a crucial aspect of modern HPC cluster construction. In addition to the previously mentioned simple NFS, larger High-Performance Computing clusters typically employ distributed file systems.

The most commonly used distributed file systems in the HPC field are open-source Lustre, commercial BeeGFS, and GPFS (IBM Spectrum Scale) file systems.

Let me share with you a previous architecture test we conducted.

We utilized the Seagate 5U84 storage system paired with standard servers and configured a Lustre distributed file system using a 100Gb InfiniBand network interconnect.

In the fully-configured 5U84 setup, we achieved a write performance of 5.53GB and a read performance of 7GB throughput. The test was conducted using fully-configured 12TB hard drives, resulting in a total capacity of approximately 1.3 petabytes (P).

By employing a combination of the 2U24 and two 5U84 units, we achieved a Lustre distributed file system with a read throughput of 10GB and a write throughput of 10GB. It's evident that the storage system exhibited excellent bandwidth throughput performance.

This level of storage configuration is suitable for CPU-intensive, memory bandwidth-sensitive, IO-intensive, IO latency-sensitive, and mixed workload computational tasks. Based on experience, a storage setup like this can support High-Performance Computing clusters ranging from 50 to 80 nodes.

We discussed the hardware aspect earlier, now let's move on to the software part.:

Previously, we discussed how ClusterTech assists users in building a complete High-Performance Computing cluster platform, starting from the system layer up to the cluster layer, container layer, tool layer, and application layer.

Merely assembling hardware and deploying the software platform without considering the specific requirements of the users would not meet their needs. Users often require assistance with software optimisation, software compilation, job submission, and other related tasks.

It is perfectly normal for users to have limited knowledge about the underlying hardware and software platform architecture. After all, everyone specializes in their own field, and it is neither practical nor reasonable to burden the customers with all these responsibilities.

As a software development and sales company, lusterTech can provide functional deployments from the system layer to the software cluster layer. We strive to integrate user applications with the CHSS cluster platform, optimize application layer deployments, and help users overcome the challenges of building an entire cluster. Our aim is to provide a turnkey solution that meets all the users' requirements.

CHESS product web interface

CHESS is an HPC (High-Performance Computing) cluster management platform based on the B/S architecture. It is primarily used for managing all hardware devices and cloud computing resources and integrating various user applications. It helps users coordinate the computational power of the entire HPC cluster, manage jobs, accounts, and provides reporting, timing, billing management, and monitoring functionalities.

Through the web portal login homepage, users can upload data, submit applications, such as TensorFlow, Fluent, and various other High-Performance Computing applications. The icons and applications can also be customised without the need for additional development.

In HPC clusters or supercomputing centers with hundreds or thousands of nodes, a multi-user management mode is generally adopted. Therefore, CHESS supports granular user management, including adding, deleting, modifying, and querying users. It also supports the deployment and configuration of job scheduling systems, accommodating various job scheduling systems, enabling users to schedule jobs on the cluster in the fastest and most efficient manner. Additionally, users can customise and publish the application software they need in fields such as bioinformatics, fluid dynamics, petroleum exploration, meteorology, and more on the CHESS platform.

For users with visualisation requirements, CHESS can run corresponding plotting and post-processing programs on the server to visually present the final results to end-users. It enables data uploading and secure management, and provides comprehensive monitoring of computing resources, node status, and resource utilization on the platform. It also monitors hardware and device resource usage and load on the entire cluster platform.

In addition to addressing the basic requirements of High-Performance Computing, CHESS offers additional features:

For example, it provides reporting and timing billing for the entire cluster, generating regular reports based on different users and departments to report workload and usage. It supports job workflow management, as many users have job dependencies where the next step depends on the success or failure of the previous step. CHESS supports this kind of workflow management. It also supports various types of AI applications.

ClusterTech has many successful cases and accumulated experience in implementing and delivering different application scenarios in manufacturing, scientific research, meteorology, biology, defense, and military industries.

ClusterTech can provide a complete set of solutions for HPC construction to users with different application scenarios and requirements, helping them solve challenges in High-Performance Computing applications.

 

That concludes my presentation for today. Thank you, everyone.