Speech | How To Build A High-Performance Computing Architecture For CAE Simulation?
2022-03-30

 

“Within the entire HPC architecture, users are most concerned with and prioritize application software performance. All software and hardware components should serve the applications, and the entire cluster should aim to enhance application efficiency.”

----Joint Partner in ClusterTech, Fu Hongyan

 

Introduction

In the industrial and engineering fields, the practical application of High-Performance Computing (HPC) in simulation and modeling not only reduces the number and cost of physical prototypes but also improves design quality and efficiency. It enhances the means and capabilities of enterprises to solve complex technical problems, ultimately shortening the time to market for new products, accelerating the introduction of new product varieties, and enhancing the domestic and international competitiveness of enterprises in related industries.

On March 22nd, ClusterTech, in collaboration with Dell Technologies and China Technology Co., Ltd, discussed the integration of High-Performance Computing and CAE (Computer-Aided Engineering) and shared practical guidelines for implementing HPC in the field of simulation and modeling.

We have summarized the transcript of this live event for your reference and learning. The following is the original text of the presentation:

 

PART01: Introduction to ClusterTech

First of all let’s introduce ClusterTech first:

ClusterTech was established in 2000 and operates in the fields of high-performance computing, cloud computing, big data, and artificial intelligence.

So far, ClusterTech has provided products and services to nearly a thousand enterprises, universities, research institutions, and government agencies.

Why establish ClusterTech? Let's start with an award:

The establishment of ClusterTech can be traced back to an award and the vision of its founder, Dr. Wai Mo Suen. Dr. Wai Mo Suen, the founder of ClusterTech, is a lifetime professor at the University of Washington, specializing in gravitational wave research.

In 1998, Dr. Suen led his students to demonstrate a scenario at the SC 98 conference. They submitted a computational task from a workstation at the conference venue. When they found two neutron stars approaching each other, they encrypted that portion of the computation and sent it to a supercomputing cluster in San Diego. When they discovered two neutron stars getting closer again, they sent the encrypted computational region to a supercomputing center in Berlin, Germany. The third encrypted region was sent to a supercomputing center in Jülich, Germany. This demonstration showcased the integration of resources from multiple locations to collectively compute a complex task, earning it an innovation award in 1998.

Around the same time, the United States established a High-Performance Computing Advisory Committee with the aim of promoting the wider usage of High-Performance Computing technology. In the 1990s, this committee was actively discussing the potential impact of High-Performance Computing on the future.

However, in China, many researchers were still unaware of or unable to utilize this technology. Consequently, two years later, ClusterTech was established in 2000 with the initial mission and vision of developing practical computing technology and enhancing scientific and technological productivity in China and the Asia-Pacific region.

Since its establishment, ClusterTech has set up branch offices or representative offices in several major cities across China. ClusterTech (Beijing) Limited is the flagship subsidiary of the group in mainland China, focusing on high-performance computing, meteorology and environmental protection, integrated simulation design, and other products and solutions.

Over the past two decades, ClusterTech has been dedicated to the field of computing technology, striving to unleash the power of HPC across various industries.

 

PART02: Ubiquitous High-Performance Computing

The scientific research methods include theoretical research, experimental research, and computational research. With the development of computer technology over the past few decades, computational research has also become an indispensable and important means in scientific research and production. High-performance computing is pervasive across various industries:

For example, in engineering simulation, it is used in aircraft design, automobile design, and chip design;

In the field of meteorology and environmental protection, it is used in weather forecasting, climate simulation, and environmental pollution prediction;

In fundamental disciplines, it is applied in molecular dynamics, structural mechanics, and fluid mechanics;

In the energy sector, it is used in seismic data processing, research on new energy sources, and the design of wind turbine loads in wind power;

In the biopharmaceutical industry, it contributes to the research of targeted therapeutic drugs, protein reconstruction, and gene sequencing;

In the field of artificial intelligence, it is involved in applications such as facial recognition, speech recognition, and autonomous driving;

In the financial and securities industry, High-Performance Computing is being increasingly used to address scenarios like quantitative trading and investment simulation;

Other industries, such as animation rendering, new materials, and remote sensing technology, also rely on High-Performance Computing.

Indeed, High-Performance Computing is truly ubiquitous, pervading numerous domains.

The evolution of High-Performance Computing can be divided into three stages:

The first stage is the era of standalone machines or server clusters, where computing resources are primarily allocated manually.

The second stage is the era of HPC clusters, which can be categorized as small-scale and large-scale HPC based on user requirements. Due to the abundance of resources, dynamic scheduling of resources becomes necessary.

The third stage is the current era of cloud computing that is widely discussed. Whether it is in local environments, public clouds, or supercomputing centers, computing resources are easily accessible, similar to the convenience of using water or electricity. This shift has liberated the computing power from the hands of a few and made it accessible to the majority of people.

After discussing High-Performance Computing in depth, what benefits does it bring to us?

High-Performance Computing can reduce computation time, lower costs, increase computing scale, and enhance efficiency. By achieving this balance, it can improve enterprise efficiency and accelerate the pace of scientific research.

 

PART03 The HPC plays a significant role in CAE simulations

Let's take a look at the CAE simulation design process, which consists of three main parts:

CAD Pre-processing: This part involves 3D modeling using GPUs.

CAE Solving: The CAE solving phase is where the actual simulation calculations take place. It heavily relies on CPU and memory resources for performing numerical computations.

CAD Post-processing: The CAD post-processing stage involves examining and verifying the simulation results to ensure they meet the desired requirements.

Software tools such as Fluent and Star-CCM+ for fluid analysis, Abaqus and Ansys for structural mechanics, LS-Dyna for collision analysis, and simulation tools for fatigue, electromagnetic, and electronic design automation (EDA) are commonly used in CAE.

What are the values of using High-Performance Computing (HPC) in engineering simulation?

Solving more complex problems

For example, precise simulation of model details, larger sizes for higher accuracy, and the ability to simulate entire assemblies from individual components.

Accelerating the development process

HPC clusters have powerful processing capabilities and high computing speeds, enabling rapid completion of a large number of computational tasks. This capability helps improve design quality and efficiency in engineering development processes.

Reducing investment costs

By utilizing computational simulations, the number and costs of physical experiments can be reduced. For instance, when designing an aircraft or a car, it is not feasible to create numerous physical models for testing due to long cycles and high costs involved. Additionally, HPC enables faster simulation runs, allowing design and development teams to iterate more quickly and efficiently. This increased efficiency translates into lower costs as the time and effort required for design iterations are reduced.

Maximizing return on investment (ROI)

Centralized resource management and scheduling can improve utilization rates by efficiently allocating computing resources, avoiding scattered and fragmented usage patterns. Additionally, centralized software installation and deployment can reduce maintenance and operational costs.

Enhancing business competitiveness

Using HPC clusters for simulation design can significantly reduce time to market for products, accelerating the introduction of new products and enhancing the competitiveness of businesses in the market.

Therefore, many companies now also build HPC clusters to facilitate production and research and development. However, there are many challenges in the process of building and using clusters, which can be summarized into two categories: how to build a good cluster and how to use it effectively.

From the perspective of IT engineers:

How to integrate different CAE software based on business scenarios for solution planning? During the construction process, how to build a reliable, user-friendly, and consistently high-performance cluster system? How to effectively manage the cluster during usage? When a large number of jobs are submitted, there may be contention for computing resources. How to improve resource utilisation?

From the perspective of enterprise engineers:

Most enterprise engineers are not familiar with another operating system. How can they quickly process data and submit jobs? How can application software be rapidly deployed and run on this platform? These are the practical difficulties faced by users.

ClusterTech HPC Engineering Simulation Solution can help users address the challenges of building and effectively using HPC clusters.

 

PART04 PART04 ClusterTech’s Engineering simulation design solutions

ClusterTech's engineering simulation design solution is built upon ClusterTech's self-developed CHESS cluster management software and over 20 years of industry experience. It aims to design, customize, build, and maintain HPC cluster systems for users. The solution provides unified management of simulation data, flexible configuration of resource pooling, and integration of GPU-accelerated remote 3D visualization components. It ensures efficient and stable operation of HPC clusters, on-demand scheduling, and effective management. This solution helps manufacturing companies achieve engineering simulation design requirements with high efficiency and low cost.

The engineering simulation design solution revolves around the application as the core, with hardware and software as the foundation.

Hardware Foundation:

Analyzing user requirements, designing, and building the optimal solution that suits their specific scenarios, to achieve the best possible match between applications and hardware.

Software Foundation:

Taking into account the usage scenarios, optimising applications on-demand, achieving the best possible match between the entire cluster system and software, and developing operational workflows that align with user habits.

Application:

On the basis of providing a stable, reliable, and high-performance software and hardware platform, achieving optimal performance of application software.

Within the entire HPC architecture, the performance of application software is the primary concern and focus for users. All software and hardware components are designed to serve the applications, and the entire cluster is aimed at enhancing application efficiency.

ClusterTech's HPC end-to-end solution consists of four major modules: cluster design, cluster construction, cluster maintenance, and application services.

 

Solution Consulting and Design

In solution design and selection, ClusterTech follows principles such as high reliability, scalability, universality, data security, and visualisation.

In the solution design process, the first step is requirement analysis to understand which application software users need, the number of users, and the specific usage scenarios. Based on this information, the solution is designed, and cost evaluation is performed.

If required, ClusterTech also provides Proof of Concept (POC) testing, including performance testing and functionality testing, to jointly confirm the solution with the users.

This is a typical HPC solution design architecture. Let's use this architecture as an example to understand how the solution design can be approached:

First, let's consider the compute nodes, which are critical components of HPC systems and come in various types and quantities. The selection of CPUs for the compute nodes is particularly important. It should be based on the user's specific usage scenario, whether it is Intel processors, AMD processors, or GPUs. If Intel processors are chosen, the specific model should be selected based on the actual requirements;

Next is the network, which connects the various components of the cluster. The network selection should also be based on the specific usage scenario, such as 10 Gigabit Ethernet, InfiniBand (IB), or Omni-Path Architecture (OPA);

In terms of underlying storage, the choice of storage architecture and parallel file system depends on the application's IO requirements. Common parallel file systems include Lustre, BeeGFS, and GPFS;

Additionally, deployment management and monitoring software for the cluster can be either open-source or commercial versions;

Regarding the operating system, the choice between Windows and Linux depends on the specific requirements. In the manufacturing industry, remote 3D visualization is often used, and the specific technology used should be matched to the user's actual scenario.

 

Cluster Construction

The cluster construction process mainly includes hardware deployment, operating system deployment, installation of cluster software (management, monitoring, scheduling, reporting, and billing software), overall cluster performance optimisation testing, installation and optimisation of application software while ensuring optimal hardware performance, and finally providing documentation and training to users.

Regarding parallel file systems, ClusterTech has the capability to provide services for BeeGFS, Lustre, and GPFS.

System testing is also crucial throughout the cluster setup process. ClusterTech has a professional team of HPC engineers who can perform comprehensive performance diagnostics on the entire cluster, including benchmark performance testing such as Linpack for individual nodes, Linpack for the entire cluster, memory bandwidth, IO performance, and network performance.

Application testing includes analyzing application characteristics, selecting appropriate resources to run the application software, and matching suitable scheduling strategies.

We also provide testing reports and analysis reports to help users diagnose the cluster system.

User training includes both administrator training and user usage training. The delivered documentation includes administrator manuals, user manuals, testing reports.

 

Cluster Maintenance and Operations

In terms of operations and maintenance, HPC clusters are large and complex, forming an integrated whole. The overall reliability of the cluster relies on the normal operation and coordinated functioning of each device. Any failure in any link can potentially lead to a decrease in the performance of the entire machine. Therefore, the management and operation of the entire cluster system are particularly important.

ClusterTech has developed a comprehensive HPC management and operation system based on the CHESS High-Performance Computing management platform. It continuously monitors the operation status of the entire cluster, swiftly locates and handles faults, assists administrators in continuously optimizing scheduling strategies, simplifies operation and maintenance management, and provides features such as reporting and billing to assist users in making decisions regarding upgrades and expansion.

 

After-Sales service

In terms of after-sales service, ClusterTech can assist users in troubleshooting issues related to the operating system, file system, job scheduling system, monitoring software, and application software. Additionally, ClusterTech provides regular inspection services to ensure the stable operation of the cluster.

 

PART05 PART05 Core Product – CHESS

CHESS is a High-Performance Computing management platform developed by ClusterTech. It enables the deployment, management, monitoring, scheduling, reporting, and billing of an entire cluster system through a web browser. It can transform loosely stacked local servers into a cohesive cluster system. Additionally, if local resources are insufficient, CHESS can also leverage public cloud resources, significantly improving cluster efficiency and simplifying operation and maintenance management.

CHESS was first released in 2001 and has undergone multiple updates and optimizations over the past two decades. The current version available is 6.6.

The CHESS functional modules include more than twenty modules, including cluster deployment, job management, user management, application management, billing, reporting, process management, license management, and more.

At the hardware level, CHESS supports servers with architectures such as X86 and ARM. When local resources are insufficient, it can be expanded with public cloud resources.

At the system level, CHESS supports operating systems such as Redhat, CentOS, and Windows 2012. It also supports parallel file systems, including GPFS, Lustre, and BeeGFS;

The cluster layer is where CHESS provides its functionality, including cluster deployment, management, monitoring, scheduling, reporting, billing, hybrid cloud management, and project management;

At the container layer, if an application is packaged in a container, CHESS can directly schedule the container to execute the application;

The tool layer includes compilers, math libraries, and debugging and optimization tools.

At the topmost layer is the application layer, where CHESS can adapt to common applications such as CAE simulation, EDA simulation, meteorology, environmental protection, and artificial intelligence.

Over the past two decades, primary focus in ClusterTech has been on running users' application software on their existing hardware platforms or public cloud resources, ensuring that they run effectively and efficiently.

 

CHESS Cluster Deployment

The cluster deployment functionality in CHESS is accomplished through the management node, which enables the batch deployment of compute nodes, I/O nodes, and login nodes. It provides flexible scalability capabilities, allowing users to easily add or remove nodes as needed. Additionally, CHESS offers power management features to control the startup and shutdown of cluster nodes.

 

CHESS Web Portal

CHESS provides a graphical and desktop-style interface that presents each application in the form of icons, making it easy to navigate and operate. The interface is designed to be user-friendly and intuitive, with a simple logical flow. Users can interact with the applications by clicking on the corresponding icons.

 

CHESS security controls

The security management module is primarily used for confidential authentication in military-industrial enterprises and provides functionalities such as three-level management, project management, and classification management.

 

CHESS user management

In CHESS, you can create users within the system or integrate it with an existing user system to enable unified identity login. The user management module in CHESS provides additional functionality, including the capability to designate secondary administrators who can manage users within their respective departments.

 

Job scheduling capabilities on CHESS

Job scheduling is indeed a core functionality within a cluster, and CHESS provides support for popular schedulers such as LSF, Slurm, and PBS. Additionally, CHESS offers multiple scheduling algorithms, including fair sharing and preemption.

Furthermore, CHESS supports GPU scheduling for various configurations, including single-node single-GPU, single-node multi-GPU, and multi-node multi-GPU setups.

 

Application management capabilities on CHESS

In CHESS, the application management functionality allows administrators to edit, publish, disable, and assign permissions to application templates. Administrators have the ability to control which users or departments can use specific application templates to submit jobs. Each application template can be created by dragging and dropping standard components, and it also supports generating application templates using HTML statements.

 

Three-dimensional remote visualization on CHESS

CHESS utilizes VNC (Virtual Network Computing) or DCV (Desktop Cloud Visualization) technology to implement three-dimensional remote visualization for pre- and post-processing tasks.

 

Data management capabilities on CHESS

Through operations on the interface, data management allows for functionalities such as uploading, downloading, compressing, and extracting Linux system files. It can also integrate with file and directory access tools.

 

Resource management Capabilities on CHESS

Resource management in CHESS encompasses both cloud-based resources and local CPU and GPU resources, as well as user quotas such as storage quota management.

 

Monitoring and operations Capabilities on CHESS

CHESS's monitoring features include a visual representation of the cabinet layout, allowing users to perform actions such as powering on/off machines and viewing machine statuses. The right-side image displays an overall system monitoring graph, which includes CPU, memory, and job execution information. This graph can be displayed on a large screen as well.

 

Billing System Checking on CHESS

CHESS provides detailed statements, reports, balance and rate settings, and supports functionalities such as recharge, overdraft, and discount factors.

 

Reporting Comprehensive Insights and Information on CHESS

Cluster reports can be categorized into two main types of information: resource usage reports and job reports. These reports provide insights into resource utilization and job usage across different dimensions such as users, departments, queues, and projects.

 

License Management on CHESS

The license management functionality of CHESS is specifically developed based on the requirements of the Electronic Design Automation (EDA) industry. Licenses in the EDA industry are both expensive and complex to manage. CHESS provides license management, monitoring, reporting, pricing, and detailed billing functionalities.

Through CHESS, users can view the status and expiration dates of license servers. It can notify users when licenses are about to expire and monitor the total number of licenses, the number of licenses in use, and the users who are using them.

 

​CHESS Process Management

CHESS process management provides task management and scheduling with dependencies. It allows users to submit jobs and track progress based on a workflow dimension.

 

​CHESS Message Management

Administrators can use the message management feature to send messages to users or post messages on the homepage. Users can receive messages from administrators, job-related messages, and billing-related messages.

 

CHESS Artificial Intelligence Module

The CHESS Artificial Intelligence Module supports GPU scheduling and also supports various training models for Artificial Intelligence.

 

CHESS Hybrid Cloud Management and Scheduling

The CHESS hybrid cloud management and scheduling feature leverages public cloud resources to supplement local resources when the local cluster is under-resourced. On the public cloud, CHESS supports both reserved instances and on-demand instances, offering the flexibility of choosing between the two modes.

 

The advantage of CHESS platform

Supports Hybrid Cloud:

The integration between local clusters and public cloud allows for the rapid implementation of hybrid cloud mode.

Users can easily scale up or down their resources in the public cloud to meet the fluctuating workload requirements, ensuring optimal resource utilization and cost efficiency in the hybrid cloud environment.

Users can access computing resources on-demand without the need to wait in queues.

Private Cluster Management:

Fast and Automated Cluster Deployment

Centralised Cluster Resource Management

Intelligent Task Scheduling

Campus/Industry/Shared Operations:

Comprehensive User Registration and Management

User-Friendly Interface

Rich Reporting and Billing Information

GPU Monitoring and Scheduling:

GPU Sharing, Exclusive Mode, and Scheduling

Support GPU sharing, exclusive mode, and scheduling for single and multiple GPUs. Real-time Monitoring of GPU Load, Memory Usage, Network Traffic, and Task Status.

Support for Various Architectures:

Support for X86 and ARM64 Architecture Servers

Support for Linux, AIX, and Windows Hybrid Clusters

Support for Multiple AI Frameworks:

Support for TensorFlow, Caffe, and other AI frameworks

Container Support:

Support for Docker and Singularity containers.

 

PART06 Case Study Sharing

Construction of High-Performance Computing Cluster for China Petrochemical Corporation (Sinopec)

For this event, we are honored to invite VIP user China Petrochemical Corporation (Sinopec), to share their experiences and the value of using CHESS. In the following, there will be more detailed tweets providing a comprehensive interpretation. Without further ado, let's move on to the detailed analysis.

 

Shanghai Fanzhineng Equipment Co., Ltd.

The user originally used workstations for their computations but found the computing power to be insufficient. They subsequently adopted an HPC cluster for simulation design. The user purchased three phases of Dell hardware equipment and utilized CHESS for cluster management. The primary software applications used include Abaqus, Numeca, Ansys Fluent, and CFX.

Previously, it took the user 48 hours to run a job on their workstations, meaning that they would start the computation in the afternoon and only see the results the day after tomorrow. However, after utilizing the cluster supported by the CHESS platform, they achieved a significant improvement in efficiency. They can now submit a task in the morning and see the results before leaving work in the afternoon. The computing efficiency has increased fourfold.

 

Project: Private Cloud + HPC Construction Project for a Aircraft Design and Research Institute of Aviation Industry Corporation of China,Ltd.

This case is about a military research institute that built a private cloud + HPC platform. In the past, cloud platforms and HPC were constructed separately because their usage is opposite. Private clouds allocate a single core to multiple users for running services, while HPC requires multiple cores to handle large computations. The private cloud construction in this project used Alibaba Cloud, marking the first implementation of a private cloud combined with HPC in China.

 

High-Performance Computing Cluster Construction Project for an Aviation Industry Institute

This case is also about a military research institute. The project is divided into two phases. In the first phase, over 200 X86 servers are deployed, while in the second phase, over 300 ARM servers are added. The cluster serves not only the internal needs of the research institute but also undergoes security certification to provide computing services for aerospace systems on the Jinhang Network. CHESS (Computing Hardware and Software Services) provides the corresponding billing functionality for these services.

 

Construction of the "Shared and Collaborative" High-Performance Computing Campus-Level Platform at Xi'an Jiaotong University

This case is about Xi'an Jiaotong University's campus-level platform, which successfully implemented the "万核大平台" (Ten Thousand Cores Mega Platform) as a shared and collaborative cluster. In this project, the users of the cluster are the ones who financially contribute, and they have high demands for accurate usage billing. CHESS effectively meets the users' requirements, making significant contributions to the research outcomes of the university.

 

Operation of Supercomputing Center - Dongfang Supercomputing Cloud Inner Mongolia Super Brain Construction Project

The Mongolia Super Brain project is a public supercomputing center. It utilizes the CHESS platform for multi-level user management, a graphical submission interface, and functionality modules such as cluster monitoring and billing. This project aims to achieve fine-grained management and 24/7 remote operation of the supercomputing center. It helps users meet the diverse computational needs of different academic disciplines by offering advantages such as low cost and fast delivery.

 

CAE (Computer-Aided Engineering) is an important field in HPC (High-Performance Computing) applications. The successful implementation of HPC solutions specifically designed for CAE heavily relies on the support of numerous industry ecosystem partners. It is believed that in the future, more users will experience the benefits and value brought by High-Performance Computing.

ClusterTech will continue to make further attempts in the field of High-Performance Computing. Together, we look forward to the continued development of High-Performance Computing, aiming for even better advancements in the future.

My sharing ends here, and thank you all for listening.