[AMDGPU][Attributor] OpenMC Performance Regression on AMD GPU

[jtramm]

I have observed a performance regression in the OpenMC scientific simulation application (which uses OpenMP target offloading) on the AMD MI250 due to commit 7dbd6cd, which is part of PR #114357. The regression is not observed on NVIDIA GPUs.

This commit causes OpenMC to be 20% slower overall on a typical benchmark problem, with the most expensive kernel in the simulation being about 2x slower.

OpenMC can be installed and its performance benchmark run using the following script: https://github.com/jtramm/openmc_offloading_builder/tree/main

FOM before this commit (higher is better):

Calculation Rate (inactive)       = 239870.0 particles/second

FOM after this commit:

 Calculation Rate (inactive)       = 192707.0 particles/second

Main kernel timing before this commit:

     XS lookups (Fuel)             = 1.8720e+01 seconds

Main kernel after this commit:

     XS lookups (Fuel)             = 3.5231e+01 seconds

Rocprof shows similar slowdown for this kernel, which it lists as:

__omp_offloading_25_7c638f81__ZN6openmc32process_calculate_xs_events_fuelEv_l256.kd

[llvmbot]

@llvm/issue-subscribers-backend-amdgpu

Author: John Tramm (jtramm)

I have observed a performance regression in the OpenMC scientific simulation application (which uses OpenMP target offloading) on the AMD MI250 due to commit https://github.com/llvm/llvm-project/commit/7dbd6cd2946ec3a9b4ad2dfd7ead177baac15bd7, which is part of PR #114357. The regression is not observed on NVIDIA GPUs.

This commit causes OpenMC to be 20% slower overall on a typical benchmark problem, with the most expensive kernel in the simulation being about 2x slower.

OpenMC can be installed and its performance benchmark run using the following script: https://github.com/jtramm/openmc_offloading_builder/tree/main

FOM before this commit (higher is better):

Calculation Rate (inactive)       = 239870.0 particles/second

FOM after this commit:

 Calculation Rate (inactive)       = 192707.0 particles/second

Main kernel timing before this commit:

     XS lookups (Fuel)             = 1.8720e+01 seconds

Main kernel after this commit:

     XS lookups (Fuel)             = 3.5231e+01 seconds

Rocprof shows similar slowdown for this kernel, which it lists as:

__omp_offloading_25_7c638f81__ZN6openmc32process_calculate_xs_events_fuelEv_l256.kd

[llvmbot]

@llvm/issue-subscribers-offload

Author: John Tramm (jtramm)

I have observed a performance regression in the OpenMC scientific simulation application (which uses OpenMP target offloading) on the AMD MI250 due to commit https://github.com/llvm/llvm-project/commit/7dbd6cd2946ec3a9b4ad2dfd7ead177baac15bd7, which is part of PR #114357. The regression is not observed on NVIDIA GPUs.

This commit causes OpenMC to be 20% slower overall on a typical benchmark problem, with the most expensive kernel in the simulation being about 2x slower.

OpenMC can be installed and its performance benchmark run using the following script: https://github.com/jtramm/openmc_offloading_builder/tree/main

FOM before this commit (higher is better):

Calculation Rate (inactive)       = 239870.0 particles/second

FOM after this commit:

 Calculation Rate (inactive)       = 192707.0 particles/second

Main kernel timing before this commit:

     XS lookups (Fuel)             = 1.8720e+01 seconds

Main kernel after this commit:

     XS lookups (Fuel)             = 3.5231e+01 seconds

Rocprof shows similar slowdown for this kernel, which it lists as:

__omp_offloading_25_7c638f81__ZN6openmc32process_calculate_xs_events_fuelEv_l256.kd

[shiltian]

The commit under discussion actually produces the correct behavior. I compared the IRs with and without the commit, and the only difference was that several functions had the attribute "amdgpu-waves-per-eu"="4,8" without the commit, whereas they did not with the commit.

This difference occurred because the values 4,8 were mistakenly added in the first run of AMDGPUAttributor due to the use of intermediate values. After the fix introduced in the commit, the pair was corrected to 1,8, which matches the default value, so the attribute was no longer added to those functions.

Interestingly, the incorrect values 4,8 performed better for this specific case. However, performance was worse with a newer, unreleased version of ROCm.

This raises the question: what is the best way to choose the amdgpu-waves-per-eu value? We still need to find an optimal strategy.