Engineering Code Scales Across 200,000 Cores on Cray Super

Teams at Saudi Aramco using the Shaheen II at King Abdullah University of Science and Technology (KAUST) supercomputer have managed to scale ANSYS Fluent across 200,000 cores, marking top-end scaling for the commercial engineering code.

The news last year of a code scalability effort that topped out at 36,000 cores on the Blue Waters machine at the National Center for Supercomputing Applications (NCSA) was impressive. That was big news for ANSYS and NCSA, but also a major milestone for Cray. Just as Blue Waters is a Cray system, albeit one at the outer reaches of its lifespan (it was installed in 2012), so is the KAUST machine.

ANSYS, Saudi Aramco and teams from KAUST sped up a complex simulation of a separation vessel from several weeks to an overnight run. This simulation is critical to all oil and gas production facilities as it reduces design development time and better predicts equipment performance under varying operational conditions. Saudi Aramco will apply this technology to make more-informed, timely decisions to retrofit separation vessels to optimize operation throughout an oil field’s lifetime.

If you’ve ever seen a rocket launch, flown on an airplane, driven a car, used a computer, touched a mobile device, crossed a bridge, or put on wearable technology, chances are you’ve used a product where ANSYS software played a critical role in its creation. One of the limitations for engineering at scale were hard barriers for code scalability—something that ANSYS and its hardware and HPC centers have worked to overcome in this and other scalability efforts.

“Today’s regulatory requirements and market expectations mean that manufacturers must develop products that are cleaner, safer, more efficient and more reliable,” said Wim Slagter, director of HPC and cloud alliances at ANSYS. “To reach such targets, designers and engineers must understand product performance with higher accuracy than ever before – especially for separation technologies, where an improved separation performance can immediately increase the efficiency and profitability of an oil field. The supercomputing collaboration between ANSYS, Saudi Aramco and KSL enabled enhanced insight in complex gas, water and crude-oil flows inside a separation vessel, which include liquid free-surface, phase mixing and droplets settling phenomena.”

“Our oil and gas facilities are among the largest in the world. We selected a complex representative application – a multiphase gravity separation vessel – to confirm the value of HPC in reducing turnover time, which is critical to our industry,” said Ehab Elsaadawy, computational modeling specialist and oil treatment team leader at Saudi Aramco’s Research and Development Center. “By working with strategic partner, KAUST, we can now run these complex simulations in one day instead of weeks.”

KSL’s Shaheen II supercomputer is a Cray system composed of 6,174 nodes representing 197,568 processor cores tightly integrated with a richly layered memory hierarchy and interconnection network.

“Multiphase problems are complex and require multiple global synchronizations, making them harder to scale than single phase laminar or turbulent flow simulation. Unstructured mesh and complex geometry add further complexity,” said Jysoo Lee, director, KAUST Supercomputing Core Lab. “Our scalability tests are not just designed for the sake of obtaining scalability at scale. This was a typical Aramco separation vessel with typical operation conditions, and larger core counts are added to reduce the time to solution. ANSYS provides a viable tool for Saudi Aramco to solve their design and analysis problems at full capacity of Shaheen.”