A sequential DO loop executed by one thread of a target parallel region runs only some of its iterations (-O2)

Legality: not independently confirmed. no independent review yet How this is checked

Same issue as cce19-target-parallel-guarded-loop; this program is another reproducer of it.

Affected variants

Compiler	MI300A gfx942 unified memory (XNACK 1)	MI300A gfx942 copy (XNACK 0)	MI250X gfx90a copy (XNACK 0)	MI300X gfx942 copy (XNACK 0)	MI350X gfx950 copy (XNACK 0)	A100 sm_80 separate memory	A100 sm_80 managed memory
AFAR 24.3.0	behaves incorrectly	behaves incorrectly	behaves incorrectly	behaves incorrectly	behaves incorrectly	n/a	n/a
CCE 19.0.0	behaves incorrectly	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
CCE 20.0.2	behaves incorrectly	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
CCE 21.0.3	behaves incorrectly	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
CCE 21.0.4	behaves incorrectly	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
NVHPC 25.11	n/a	n/a	n/a	n/a	n/a	correct	correct
NVHPC 26.9	n/a	n/a	n/a	n/a	n/a	correct	correct
GCC 16.2.0	correct	correct	correct	correct	correct	correct	n/a

"behaves incorrectly" / "not supported": this issue, confirmed in up to four environments (another build directory, another node, a repeat). "fails another way": a different failure, not counted as this issue. "unknown": not run there, or the runs disagree.

Messages

Result	Compiler and GPU	Message
behaves incorrectly	AFAR 24.3.0, MI300A unified AFAR 24.3.0, MI300A copy AFAR 24.3.0, MI250X AFAR 24.3.0, MI300X AFAR 24.3.0, MI350X CCE 19.0.0, MI300A copy CCE 19.0.0, MI250X CCE 20.0.2, MI300A copy CCE 20.0.2, MI250X CCE 21.0.3, MI300A copy CCE 21.0.3, MI250X CCE 21.0.4, MI250X	show threads=256 iterations run by threads 0-3: 1 1 1 1 threads that did not run all 64 iterations: 256 (expected 0)
behaves incorrectly	CCE 19.0.0, MI300A unified CCE 20.0.2, MI300A unified CCE 21.0.4, MI300A unified	show threads=256 iterations run by threads 0-3: 37 1 1 1 threads that did not run all 64 iterations: 256 (expected 0)
behaves incorrectly	CCE 21.0.3, MI300A unified	show threads=256 iterations run by threads 0-3: 38 1 1 1 threads that did not run all 64 iterations: 256 (expected 0)
behaves incorrectly	CCE 21.0.4, MI300A copy	show threads=256 iterations run by threads 0-3: 2 2 2 2 threads that did not run all 64 iterations: 256 (expected 0)

Reproduce

AFAR 24.3.0, MI300A unified

$AFAR/bin/amdflang -fopenmp --offload-arch=gfx942 -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 19.0.0, MI300A unified

module load cce/19.0.0 craype-accel-amd-gfx942 rocm/6.4.3
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 20.0.2, MI300A unified

module load cce/20.0.2 craype-accel-amd-gfx942 rocm/6.4.3
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 21.0.3, MI300A unified

module load cce/21.0.3 craype-accel-amd-gfx942 rocm/7.2.0
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 21.0.4, MI300A unified

module load cce/21.0.4 craype-accel-amd-gfx942 rocm/7.2.0
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

NVHPC 25.11, A100 separate (runs correctly)

export LD_LIBRARY_PATH=$NVHPC_25_11/compilers/lib
nvfortran -mp=gpu -gpu=cc80 -gpu=mem:separate -O2 repro.f90 -o repro
OMP_TARGET_OFFLOAD=mandatory ./repro

Program (44 lines)

! loop bind(thread) in a parallel region in a target region: the binding
! region is not defined and the binding thread set is the encountering
! thread (5.2 §11.7.1 p.257-258), so every thread of the team runs all n
! logical iterations ("Each logical iteration is executed once per
! instance of the loop region that is encountered by exactly one thread
! that is a member of the binding thread set", 5.2 §11.7 p.257). Each
! iteration writes its own element hit(i, thread + 1): no data race even
! though the iterations may execute concurrently.
! Expected: every thread ran all 64 iterations (0 short).
! Observed (MI300A, HSA_XNACK 0 and 1, 3 runs each): AFAR flang: 256
! threads, 1 iteration each for threads 0-3 (the team shares the
! iterations, as bind(parallel) would); CCE 19, 20, 21 ftn: 37-38, 1, 1, 1
! at HSA_XNACK=1 and 1 or 2 per thread at HSA_XNACK=0; all 256 threads
! short. The C form (AMD clang) is correct: 64 per thread.
! : on CCE the same loop without the loop
! directive fails too, and the program is correct at -O0: CCE's failure is
! the sequential-loop defect of cce19-target-parallel-guarded-loop.f90, not
! a bind(thread) defect. On AFAR it is a bind(thread) defect.
program bind_thread_repro
  use omp_lib
  implicit none
  integer, parameter :: n = 64
  integer :: hit(n, 1024), nthr, tid, i, k, short
  hit = 0
  nthr = -1
  !$omp target map(tofrom: hit, nthr)
    !$omp parallel private(tid)
      tid = omp_get_thread_num()
      if (tid == 0) nthr = omp_get_num_threads()
      !$omp loop bind(thread)
        do i = 1, n
          if (tid < 1024) hit(i, tid + 1) = hit(i, tid + 1) + 1
        end do
      !$omp end loop
    !$omp end parallel
  !$omp end target
  short = 0
  do k = 1, min(nthr, 1024)
    if (sum(hit(:, k)) /= n) short = short + 1
  end do
  print '(a,i0,a,4(1x,i0))', 'threads=', nthr, ' iterations run by threads 0-3:', &
    (sum(hit(:, k)), k = 1, 4)
  print '(a,i0,a)', 'threads that did not run all 64 iterations: ', short, ' (expected 0)'
end program bind_thread_repro

Source in the repository

Something to add? Dispute its legality · Report it fixed in a newer release · Report a mistake on this page

Expected and observed

Observed where it behaves incorrectly

threads=256 iterations run by threads 0-3: 1 1 1 1
threads that did not run all 64 iterations: 256 (expected 0)

What the specification requires

The specification text this program relies on and the compiler is expected to support.

OpenMP 5.2 §11.7.1 p.257-258

OpenMP 5.2 §11.7 p.257

Each logical iteration is executed once per instance of the loop region that is encountered by exactly one thread that is a member of the binding thread set

Evidence

Same compiler, one change

the loop directive removed (each thread runs a plain DO loop): AFAR 24.3.0: correct; CCE 19.0.0: behaves incorrectly; CCE 20.0.2: behaves incorrectly; CCE 21.0.3: behaves incorrectly; CCE 21.0.4: behaves incorrectly

Control program

program bind_thread_repro
  use omp_lib
  implicit none
  integer, parameter :: n = 64
  integer :: hit(n, 1024), nthr, tid, i, k, short
  hit = 0
  nthr = -1
  !$omp target map(tofrom: hit, nthr)
    !$omp parallel private(tid)
      tid = omp_get_thread_num()
      if (tid == 0) nthr = omp_get_num_threads()
        do i = 1, n
          if (tid < 1024) hit(i, tid + 1) = hit(i, tid + 1) + 1
        end do
    !$omp end parallel
  !$omp end target
  short = 0
  do k = 1, min(nthr, 1024)
    if (sum(hit(:, k)) /= n) short = short + 1
  end do
  print '(a,i0,a,4(1x,i0))', 'threads=', nthr, ' iterations run by threads 0-3:', &
    (sum(hit(:, k)), k = 1, 4)
  print '(a,i0,a)', 'threads that did not run all 64 iterations: ', short, ' (expected 0)'
end program bind_thread_repro

Other compilers run it correctly

NVHPC 25.11, NVHPC 26.9, GCC 16.2.0