A sequential DO loop executed by one thread of a target parallel region runs only some of its iterations (-O2)

Legality: independently confirmed. 2 independent readings find the program legal and the result required. How this is checked

Affected variants

Compiler	MI300A gfx942 unified memory (XNACK 1)	MI300A gfx942 copy (XNACK 0)	MI250X gfx90a copy (XNACK 0)	MI300X gfx942 copy (XNACK 0)	MI350X gfx950 copy (XNACK 0)	A100 sm_80 separate memory	A100 sm_80 managed memory
AFAR 24.3.0	correct	correct	correct	correct	correct	n/a	n/a
CCE 19.0.0	behaves incorrectly	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
CCE 20.0.2	behaves incorrectly	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
CCE 21.0.3	behaves incorrectly	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
CCE 21.0.4	behaves incorrectly	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
NVHPC 25.11	n/a	n/a	n/a	n/a	n/a	correct	correct
NVHPC 26.9	n/a	n/a	n/a	n/a	n/a	correct	correct
GCC 16.2.0	correct	correct	correct	correct	correct	correct	n/a

"behaves incorrectly" / "not supported": this issue, confirmed in up to four environments (another build directory, another node, a repeat). "fails another way": a different failure, not counted as this issue. "unknown": not run there, or the runs disagree.

Messages

Result	Compiler and GPU	Message
behaves incorrectly	CCE 19.0.0, MI300A unified CCE 19.0.0, MI300A copy CCE 19.0.0, MI250X CCE 20.0.2, MI300A unified CCE 20.0.2, MI300A copy CCE 20.0.2, MI250X CCE 21.0.3, MI300A unified CCE 21.0.3, MI300A copy CCE 21.0.3, MI250X CCE 21.0.4, MI300A unified CCE 21.0.4, MI300A copy CCE 21.0.4, MI250X	show base_cond bad= 63

Reproduce

CCE 19.0.0, MI300A unified

module load cce/19.0.0 craype-accel-amd-gfx942 rocm/6.4.3
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 20.0.2, MI300A unified

module load cce/20.0.2 craype-accel-amd-gfx942 rocm/6.4.3
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 21.0.3, MI300A unified

module load cce/21.0.3 craype-accel-amd-gfx942 rocm/7.2.0
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 21.0.4, MI300A unified

module load cce/21.0.4 craype-accel-amd-gfx942 rocm/7.2.0
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

AFAR 24.3.0, MI300A unified (runs correctly)

$AFAR/bin/amdflang -fopenmp --offload-arch=gfx942 -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

Program (30 lines)

program t
  use omp_lib, only: omp_get_thread_num
  implicit none
  integer, parameter :: n = 64
  integer, allocatable :: a(:), b(:)
  integer :: ref(n), i, nbad
  allocate(a(n), b(n))
  do i = 1, n
    a(i) = i; b(i) = 0; ref(i) = 2*i + 1
  end do
  !$omp target parallel map(to: a) map(from: b) 
    if (omp_get_thread_num() == 0) then
      do i = 1, n
        b(i) = 2*a(i) + 1
      end do
    end if
  !$omp end target parallel
  nbad = count(b /= ref)
  print *, 'base_cond bad=', nbad
end program
! spec (requirement): Every thread of the team created by a parallel construct executes the region, and each thread
!   may follow a different path of statements.
! "Each thread can execute a path of statements that is different from that of the other threads." (5.2 §10.1 p.223
!   l.27-28)
! spec (requirement): The thread with thread number zero is the primary thread and takes part in executing the region
!   like every other thread.
! "All threads in the new team, including the primary thread, execute the region." (5.2 §10.1 p.223 l.16-17)
! spec (this program): Only thread 0 enters the guarded branch and runs the sequential DO loop of 64 iterations, so
!   base-language semantics require all 64 elements of b to be written; the spec adds no rule that cuts a sequential
!   loop short. At -O2 the compiler runs only some of the iterations.

Source in the repository

Something to add? Dispute its legality · Report it fixed in a newer release · Report a mistake on this page

Expected and observed

Expected

base_cond bad= 0

Observed where it behaves incorrectly

 base_cond bad= 63

What the specification requires

The specification text this program relies on and the compiler is expected to support.

OpenMP 5.2 §10.1 p.223 l.27-28

Each thread can execute a path of statements that is different from that of the other threads.

OpenMP 5.2 §10.1 p.223 l.16-17

All threads in the new team, including the primary thread, execute the region.

Evidence

Same compiler, one change

compiled at -O0 instead of -O2 (-O0): AFAR 24.3.0: correct; CCE 19.0.0: correct; CCE 20.0.2: correct; CCE 21.0.3: correct; CCE 21.0.4: correct

Control program

program t
  use omp_lib, only: omp_get_thread_num
  implicit none
  integer, parameter :: n = 64
  integer, allocatable :: a(:), b(:)
  integer :: ref(n), i, nbad
  allocate(a(n), b(n))
  do i = 1, n
    a(i) = i; b(i) = 0; ref(i) = 2*i + 1
  end do
  !$omp target parallel map(to: a) map(from: b)
    if (omp_get_thread_num() == 0) then
      do i = 1, n
        b(i) = 2*a(i) + 1
      end do
    end if
  !$omp end target parallel
  nbad = count(b /= ref)
  print *, 'base_cond bad=', nbad
end program

Other compilers run it correctly

AFAR 24.3.0, NVHPC 25.11, NVHPC 26.9, GCC 16.2.0