ftn: target update from(b(lo:hi)) inside a host parallel region moves no data in copy mode (HSA_XNACK=0)

Legality: independently confirmed. 2 independent readings find the program legal and the result required. How this is checked

Affected variants

Compiler	MI300A gfx942 unified memory (XNACK 1)	MI300A gfx942 copy (XNACK 0)	MI250X gfx90a copy (XNACK 0)	MI300X gfx942 copy (XNACK 0)	MI350X gfx950 copy (XNACK 0)	A100 sm_80 separate memory	A100 sm_80 managed memory
AFAR 24.3.0	correct	unknown varies between runs	correct	unknown varies between runs	correct	n/a	n/a
CCE 19.0.0	correct	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
CCE 20.0.2	correct	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
CCE 21.0.3	correct	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
CCE 21.0.4	correct	behaves incorrectly	behaves incorrectly	unknown not run	unknown not run	n/a	n/a
NVHPC 25.11	n/a	n/a	n/a	n/a	n/a	correct	correct
NVHPC 26.9	n/a	n/a	n/a	n/a	n/a	correct	correct
GCC 16.2.0	correct	correct	correct	correct	correct	correct	n/a

"behaves incorrectly" / "not supported": this issue, confirmed in up to four environments (another build directory, another node, a repeat). "fails another way": a different failure, not counted as this issue. "unknown": not run there, or the runs disagree.

Messages

Result	Compiler and GPU	Message
behaves incorrectly	CCE 19.0.0, MI300A copy CCE 19.0.0, MI250X CCE 20.0.2, MI300A copy CCE 20.0.2, MI250X CCE 21.0.3, MI300A copy CCE 21.0.3, MI250X CCE 21.0.4, MI300A copy CCE 21.0.4, MI250X	show bad= 64

Reproduce

CCE 19.0.0, MI300A copy

module load cce/19.0.0 craype-accel-amd-gfx942 rocm/6.4.3
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=0 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 20.0.2, MI300A copy

module load cce/20.0.2 craype-accel-amd-gfx942 rocm/6.4.3
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=0 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 21.0.3, MI300A copy

module load cce/21.0.3 craype-accel-amd-gfx942 rocm/7.2.0
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=0 OMP_TARGET_OFFLOAD=mandatory ./repro

CCE 21.0.4, MI300A copy

module load cce/21.0.4 craype-accel-amd-gfx942 rocm/7.2.0
module unload cray-mpich
ftn -fopenmp -O2 repro.f90 -o repro
HSA_XNACK=0 OMP_TARGET_OFFLOAD=mandatory ./repro

AFAR 24.3.0, MI300A unified (runs correctly)

$AFAR/bin/amdflang -fopenmp --offload-arch=gfx942 -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro

Program (46 lines)

! scope: host+device: multi-threaded offload (host threads each offload their part of an array)
! Host+device pattern (found while making the multi-threaded
! offload programs conforming): b is mapped whole by target enter data and
! written on the device; then four host threads (parallel do,
! num_threads(4)) each copy their quarter back with
! target update from(b(lo:hi)). The sections are disjoint and present
! (5.2 §5.9.2: each list item is updated from its corresponding list
! item), so every element comes back. Expected: bad= 0.
! Observed (tuolumne, MI300A, 2026-10-21, 5 runs each, -O2,
! OMP_TARGET_OFFLOAD=mandatory), at HSA_XNACK=0 only (correct at 1):
! - CCE 19.0.0, 20.0.2, 21.0.3: bad= 64 on every run: no update inside the
!   parallel region moves data; the same with num_threads(1) and with
!   fixed-size arrays. Without the parallel region the updates work. A
!   program with target update to(a(lo:hi)) in the parallel region and b
!   copied back by target exit data fails the same way (bad= 64), so the
!   to direction is affected as well.
! - AFAR 24.0.0git: bad= 16 in 3 of 5 runs (one quarter lost); correct
!   with num_threads(1) and with fixed-size arrays (concurrent updates of
!   disjoint sections of one allocatable).
! AFAR: amdflang -fopenmp --offload-arch=gfx942 -O2
! CCE:  ftn -fopenmp -O2 (module load cce/<v> craype-accel-amd-gfx942 rocm)
! Run:  OMP_TARGET_OFFLOAD=mandatory HSA_XNACK=0 ./a.out
program p
  implicit none
  integer, parameter :: n = 64
  integer, allocatable :: b(:)
  integer :: i, c, lo, hi, bad
  allocate(b(n))
  b = 0
  !$omp target enter data map(to: b)
  !$omp target
    do i = 1, n
      b(i) = 2*i + 1
    end do
  !$omp end target
  !$omp parallel do num_threads(4) private(lo, hi)
  do c = 1, 4
    lo = (c - 1)*16 + 1
    hi = c*16
    !$omp target update from(b(lo:hi))
  end do
  !$omp end parallel do
  !$omp target exit data map(release: b)
  bad = count(b /= [(2*i + 1, i = 1, n)])
  print *, 'bad=', bad
end program p

Source in the repository

Something to add? Dispute its legality · Report it fixed in a newer release · Report a mistake on this page

Expected and observed

Expected

bad= 0

Observed where it behaves incorrectly

 bad= 64

What the specification requires

The specification text this program relies on and the compiler is expected to support.

OpenMP 5.2 §5.9.2

Evidence

Same compiler, one change

the parallel do directive removed (the four updates run one after another): AFAR 24.3.0: correct; CCE 19.0.0: correct; CCE 20.0.2: correct; CCE 21.0.3: correct; CCE 21.0.4: correct

Control program

program p
  implicit none
  integer, parameter :: n = 64
  integer, allocatable :: b(:)
  integer :: i, c, lo, hi, bad
  allocate(b(n))
  b = 0
  !$omp target enter data map(to: b)
  !$omp target
    do i = 1, n
      b(i) = 2*i + 1
    end do
  !$omp end target
  do c = 1, 4
    lo = (c - 1)*16 + 1
    hi = c*16
    !$omp target update from(b(lo:hi))
  end do
  !$omp target exit data map(release: b)
  bad = count(b /= [(2*i + 1, i = 1, n)])
  print *, 'bad=', bad
end program p

Other compilers run it correctly

AFAR 24.3.0, NVHPC 25.11, NVHPC 26.9, GCC 16.2.0