OffTarget

Observed issues › stress/cce-lastprivate-num-threads-collapse-simd

Appears to behave incorrectlyWrong answer · Fortran · OpenMP · Open

Lastprivate on target teams distribute parallel do simd collapse(2) with num_threads gives a varying wrong value

Legality: independently confirmed. 2 independent readings find the program legal and the result required. How this is checked

Affected variants

CompilerMI300A gfx942
unified memory (XNACK 1)
MI300A gfx942
copy (XNACK 0)
MI250X gfx90a
copy (XNACK 0)
MI300X gfx942
copy (XNACK 0)
MI350X gfx950
copy (XNACK 0)
A100 sm_80
separate memory
A100 sm_80
managed memory
AFAR 24.3.0correctcorrectcorrectfails another wayfails another wayn/an/a
CCE 19.0.0behaves incorrectlybehaves incorrectlybehaves incorrectlyunknown
not run
unknown
not run
n/an/a
CCE 20.0.2behaves incorrectlyunknown
varies between runs
behaves incorrectlyunknown
not run
unknown
not run
n/an/a
CCE 21.0.3behaves incorrectlybehaves incorrectlybehaves incorrectlyunknown
not run
unknown
not run
n/an/a
CCE 21.0.4behaves incorrectlybehaves incorrectlybehaves incorrectlyunknown
not run
unknown
not run
n/an/a
NVHPC 25.11n/an/an/an/an/abehaves incorrectlybehaves incorrectly
NVHPC 26.9n/an/an/an/an/abehaves incorrectlybehaves incorrectly
GCC 16.2.0fails another wayfails another waycorrectfails another wayfails another waycorrectn/a

"behaves incorrectly" / "not supported": this issue, confirmed in up to four environments (another build directory, another node, a repeat). "fails another way": a different failure, not counted as this issue. "unknown": not run there, or the runs disagree.

Messages

ResultCompiler and GPUMessage
behaves incorrectlyCCE 19.0.0, MI300A unified
CCE 19.0.0, MI300A copy
CCE 19.0.0, MI250X
CCE 20.0.2, MI250X
CCE 21.0.3, MI300A copy
CCE 21.0.3, MI250X
CCE 21.0.4, MI300A copy
CCE 21.0.4, MI250X
NVHPC 25.11, A100 separate
NVHPC 25.11, A100 managed
NVHPC 26.9, A100 separate
NVHPC 26.9, A100 managed
show
lp= 0
behaves incorrectlyCCE 20.0.2, MI300A unified
show
lp= 104
behaves incorrectlyCCE 21.0.3, MI300A unified
show
lp= 48
behaves incorrectlyCCE 21.0.4, MI300A unified
show
lp= 44
fails another wayAFAR 24.3.0, MI300X
AFAR 24.3.0, MI350X
show
error: loc("cce-lastprivate-num-threads-collapse-simd.f90":5:9): /__w/TheRock/TheRock/compiler/amd-llvm/flang/lib/Lower/OpenMP/DataSharingProcessor.cpp:490: not yet implemented: lastprivate clause in 
LLVM ERROR: aborting
fails another wayGCC 16.2.0, MI300A unified
GCC 16.2.0, MI300A copy
GCC 16.2.0, MI300X
GCC 16.2.0, MI350X
show
lp= 200

Reproduce

CCE 19.0.0, MI300A unified

module load cce/19.0.0 craype-accel-amd-gfx942 rocm/6.4.3
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 20.0.2, MI300A unified

module load cce/20.0.2 craype-accel-amd-gfx942 rocm/6.4.3
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 21.0.3, MI300A unified

module load cce/21.0.3 craype-accel-amd-gfx942 rocm/7.2.0
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 21.0.4, MI300A unified

module load cce/21.0.4 craype-accel-amd-gfx942 rocm/7.2.0
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

NVHPC 25.11, A100 separate

export LD_LIBRARY_PATH=$NVHPC_25_11/compilers/lib
nvfortran -mp=gpu -gpu=cc80 -gpu=mem:separate -O2 repro.f90 -o repro
OMP_TARGET_OFFLOAD=mandatory ./repro

NVHPC 26.9, A100 separate

export LD_LIBRARY_PATH=$NVHPC_26_9/compilers/lib
nvfortran -mp=gpu -gpu=cc80 -gpu=mem:separate -O2 repro.f90 -o repro
OMP_TARGET_OFFLOAD=mandatory ./repro

AFAR 24.3.0, MI300A unified (runs correctly)

$AFAR/bin/amdflang -fopenmp --offload-arch=gfx942 -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro
Program (62 lines)
! CCE 19, 20, 21: lastprivate on a composite target teams distribute
! parallel do is wrong, and varies between runs, when the parallel leaf
! runs with num_threads(2) -- or with if(parallel: .false.) and no
! num_threads. Neither collapse nor simd is needed (fix:
! this 10-line program replaces the collapse(2)/simd/num_threads(32)
! reproducer reduced from the 14-clause stress program; the
! file keeps its name so its links stay).
! Spec: lp is assigned in every iteration, so "the value of each new list
! item from the sequentially last iteration of the associated loops ... is
! assigned to the original list item" (OpenMP 5.2 §5.4.5 p.115 l.12-16),
! and on a combined target construct the list item is also map(tofrom)
! (§17.2 p.340 l.27-30). Race-free: i is the loop's iteration variable.
! Expected: lp= 200
! Observed (Tuolumne MI300A, -O2, OMP_TARGET_OFFLOAD=mandatory, 10 runs per
! cell, 2026-10-26; data/evidence/gate_probe/logs/lastprivate_simdred.txt, programs
! in data/evidence/gate_probe/lastprivate/):
! - this program (nt2): CCE 19, 20 and 21 at HSA_XNACK 0 and 1: lp= 0, 61,
!   125 or 189, varying from run to run (200 in none of the 60 runs; this
!   file as committed, which drops nt2's unused lpar: the same, 0 of 60,
!   data/evidence/gate_probe/logs/reduce.txt lpfind);
! - if(parallel: lpar) with lpar = .false. instead of num_threads(2)
!   (ifpar): the same values on all six cells;
! - with collapse(2) (collapsent2) or simd (simdnt2) added: the same;
! - control, the same directive without num_threads and if (nont): lp= 200
!   in 60 of 60 runs on CCE 19/20/21, both modes.
! Mechanism: CCE 19/20/21 warn at compile time, on this file
! and on the if(parallel: .false.) form but not on the control:
!   ftn-7256 ftn: WARNING P, File = lp.f90, Line = <the directive>
!   An OpenMP parallel construct in a target region is limited to a single
!   thread.
! (data/evidence/gate_probe/logs/lastprivate_compile.txt). Both triggers make CCE
! run the parallel leaf with one thread; so the defect is that lastprivate
! is lost when CCE serializes the parallel leaf of a composite distribute
! parallel do. The warning says nothing about lastprivate: it is not a
! diagnostic of the wrong value, and one thread must still leave lp = 200.
! The earlier reproducer (num_threads(32) on target teams distribute
! parallel do simd collapse(2) over 16 x 8 iterations: lp= 0, 20, 64 or
! 100, correct with num_teams(1) or num_teams(3),
! data/reports/fuzz-stress-audit/r7-*.f90) is this defect; so are the
! lastprivate cells of the reproduction gate's flagged cases
! (data/reports/gate-queue.md: fuzz and evidence programs with
! lastprivate on distribute parallel do with num_threads(2..8) or
! if(parallel: .false.), and the 14-clause stress program clauses-omp-16).
! Another family: AFAR amdflang gives lp= 200 in 20 of 20 runs (XNACK 1 and
! 0) on the simd form (simdnt2); on this form it stops with "not yet
! implemented: lastprivate clause in constructs other than
! simd/worksharing-loop/taskloop" (a compile failure, not a value).
! NVHPC documents lastprivate on distribute as unsupported (HPC Compilers
! User's Guide 25.11 §7.4, 2.9.4), so it is no witness here.
! CCE: ftn -fopenmp -O2 (module load cce/<v> craype-accel-amd-gfx942 rocm)
! Run: OMP_TARGET_OFFLOAD=mandatory HSA_XNACK=0 (or 1) ./a.out
program p
  implicit none
  integer :: i, lp
  lp = 0
  !$omp target teams distribute parallel do lastprivate(lp) num_threads(2)
  do i = 1, 200
    lp = i
  end do
  !$omp end target teams distribute parallel do
  print '(a,i0)', 'lp= ', lp
end program p

Source in the repository

Something to add? Dispute its legality · Report it fixed in a newer release · Report a mistake on this page

Expected and observed

Expected

lp= 200

Observed where it behaves incorrectly

lp= 0

What the specification requires

The specification text this program relies on and the compiler is expected to support.

OpenMP 5.2 §5.4.5 p.115 l.12-16

the value of each new list item from the sequentially last iteration of the associated loops ... is assigned to the original list item

OpenMP 5.2 §17.2 p.340 l.27-30

Evidence

Same compiler, one change

  • num_threads(32) removed: AFAR 24.3.0: correct; CCE 19.0.0: correct; CCE 20.0.2: correct; CCE 21.0.3: correct; CCE 21.0.4: correct
    Control program
    program lastprivate_collapse_simd
      implicit none
      integer :: i, j, lin
      integer(8) :: lp
      lp = 0
      !$omp target teams distribute parallel do simd collapse(2) private(lin) lastprivate(lp)
      do i = 1, 16
        do j = 1, 8
          lin = (i - 1) * 8 + j
          lp = lin
        end do
      end do
      !$omp end target teams distribute parallel do simd
      print '(a,i0)', 'lp= ', lp
    end program lastprivate_collapse_simd
    

Other compilers run it correctly

AFAR 24.3.0, GCC 16.2.0