Observed issues › multithread-allocatable-section-maps
Two host threads mapping disjoint halves of an allocatable array lose one half
Legality: independently confirmed. One independent reading was uncertain; 2 adjudicating readings that weighed both find the program legal and the result required: The two host threads map disjoint sections a(lo:hi) and b(lo:hi), and the target region references only elements inside the mapped sections. The only stated restriction (whole-array reference to a partly mapped array) is not triggered, and the variable is not allocated, deallocated or reshaped meanwhile. i, lo and hi are private. How this is checked
Affected variants
| Compiler | MI300A gfx942 unified memory (XNACK 1) | MI300A gfx942 copy (XNACK 0) | MI250X gfx90a copy (XNACK 0) | MI300X gfx942 copy (XNACK 0) | MI350X gfx950 copy (XNACK 0) | A100 sm_80 separate memory | A100 sm_80 managed memory |
|---|---|---|---|---|---|---|---|
| AFAR 24.3.0 | correct | behaves incorrectly | unknown not run | behaves incorrectly | behaves incorrectly | n/a | n/a |
| CCE 19.0.0 | behaves incorrectly | behaves incorrectly | unknown not run | unknown not run | unknown not run | n/a | n/a |
| CCE 20.0.2 | behaves incorrectly | behaves incorrectly | unknown not run | unknown not run | unknown not run | n/a | n/a |
| CCE 21.0.3 | behaves incorrectly | behaves incorrectly | unknown not run | unknown not run | unknown not run | n/a | n/a |
| CCE 21.0.4 | behaves incorrectly | behaves incorrectly | unknown not run | unknown not run | unknown not run | n/a | n/a |
| NVHPC 25.11 | n/a | n/a | n/a | n/a | n/a | correct | correct |
| NVHPC 26.9 | n/a | n/a | n/a | n/a | n/a | correct | correct |
| GCC 16.2.0 | fails another way | fails another way | fails another way | fails another way | fails another way | fails another way | n/a |
"behaves incorrectly" / "not supported": this issue, confirmed in up to four environments (another build directory, another node, a repeat). "fails another way": a different failure, not counted as this issue. "unknown": not run there, or the runs disagree.
Messages
| Result | Compiler and GPU | Message |
|---|---|---|
| behaves incorrectly | AFAR 24.3.0, MI300A copy AFAR 24.3.0, MI300X AFAR 24.3.0, MI350X CCE 19.0.0, MI300A unified CCE 19.0.0, MI300A copy CCE 20.0.2, MI300A unified CCE 20.0.2, MI300A copy CCE 21.0.3, MI300A unified CCE 21.0.3, MI300A copy CCE 21.0.4, MI300A unified CCE 21.0.4, MI300A copy | showbad= 32 |
| fails another way | GCC 16.2.0, MI300A unified GCC 16.2.0, MI250X GCC 16.2.0, MI300X GCC 16.2.0, MI350X | showlibgomp: OMP_TARGET_OFFLOAD is set to MANDATORY, but device not found libgomp: OMP_TARGET_OFFLOAD is set to MANDATORY, but device not found |
| fails another way | GCC 16.2.0, MI300A copy | showlibgomp: libgomp: OMP_TARGET_OFFLOAD is set to MANDATORY, but device not found OMP_TARGET_OFFLOAD is set to MANDATORY, but device not found Program received signal SIGSEGV: Segmentation fault - invalid memory reference. Backtrace for this error: |
| fails another way | GCC 16.2.0, A100 separate | showlibgomp: OMP_TARGET_OFFLOAD is set to MANDATORY, but device not found |
Reproduce
AFAR 24.3.0, MI300A copy
$AFAR/bin/amdflang -fopenmp --offload-arch=gfx942 -O2 repro.f90 -o repro HSA_XNACK=0 OMP_TARGET_OFFLOAD=mandatory ./repro
CCE 19.0.0, MI300A unified
module load cce/19.0.0 craype-accel-amd-gfx942 rocm/6.4.3 module unload cray-mpich ftn -fopenmp -O2 repro.f90 -o repro HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro
CCE 20.0.2, MI300A unified
module load cce/20.0.2 craype-accel-amd-gfx942 rocm/6.4.3 module unload cray-mpich ftn -fopenmp -O2 repro.f90 -o repro HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro
CCE 21.0.3, MI300A unified
module load cce/21.0.3 craype-accel-amd-gfx942 rocm/7.2.0 module unload cray-mpich ftn -fopenmp -O2 repro.f90 -o repro HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro
CCE 21.0.4, MI300A unified
module load cce/21.0.4 craype-accel-amd-gfx942 rocm/7.2.0 module unload cray-mpich ftn -fopenmp -O2 repro.f90 -o repro HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro
NVHPC 25.11, A100 separate (runs correctly)
export LD_LIBRARY_PATH=$NVHPC_25_11/compilers/lib nvfortran -mp=gpu -gpu=cc80 -gpu=mem:separate -O2 repro.f90 -o repro OMP_TARGET_OFFLOAD=mandatory ./repro
Program (86 lines)
! scope: host+device: multi-threaded offload (host threads each offload their part of an array)
! Host+device pattern (minimized and re-verified): two
! host threads (parallel do, num_threads(2)) each offload one half of an
! allocatable array b with map(to: a(lo:hi)) map(from: b(lo:hi)).
!
! Conformance (OpenMP 5.2). The target region references b (and a), so b
! also has an implicit data-mapping attribute (5.2 §5.8.1, p.148:
! "References to structure elements or array elements are treated as
! references to the structure or array, respectively, for the purposes of
! determining implicit data-mapping or data-sharing attributes"). When the
! other thread's half is not yet on the device, 5.2 §5.8.3 p.154: "If a
! list item with an implicit data-mapping attribute does not have any
! corresponding storage in the device data environment prior to a task
! encountering the construct associated with the map clause, and one or
! more contiguous parts of the original storage are either list items or
! base pointers to list items that are explicitly mapped on the
! construct, only those parts of the original storage will have
! corresponding storage in the device data environment as a result of the
! map clauses on the construct." When it is, p.154: "If a single
! contiguous part of the original storage of a list item with an implicit
! data-mapping attribute has corresponding storage in the device data
! environment prior to a task encountering the construct on which the map
! clause appears, only that part of the original storage will have
! corresponding storage in the device data environment as a result of the
! map clause." Each explicit half is a distinct mappable storage block
! (p.152: "The storage for each other map clause list item becomes a
! distinct mappable storage block") that no earlier construct created, so
! no restriction of p.156 applies ("If any part of the original storage
! of a list item with an explicit data-mapping attribute has corresponding
! storage ... all of the original storage must have corresponding
! storage": no part of the other half is mapped). Every interleaving is
! defined and every half comes back. Expected: bad= 0.
!
! With three or more threads (the program: four quarters) the
! program is NOT conforming: when a construct is encountered while the
! sections of two other threads are present, p.156: "If an array appears
! as a list item in a map clause, multiple parts of the array have
! corresponding storage in the device data environment prior to a task
! encountering the construct associated with the map clause, and the
! corresponding storage for those parts was created by maps from more
! than one earlier construct, the behavior is unspecified." (5.1 §2.21.7.1
! p.354 has the same sentence; 6.0 §7.9.6 p.287 makes it a restriction:
! "If a list item in a map clause has corresponding storage in the device
! data environment, all corresponding storage must correspond to a single
! mappable storage block that was previously mapped", which this
! two-thread program also violates under 6.0 when the constructs overlap.)
! The catalog's multi-threaded programs now map the arrays whole once.
!
! Observed (tuolumne, MI300A, 2026-10-21, 5 runs each, -O2,
! OMP_TARGET_OFFLOAD=mandatory): AFAR 24.0.0git bad= 32 at HSA_XNACK=0
! (correct at 1, zero-copy); CCE 19.0.0, 20.0.2, 21.0.3 bad= 32 at both.
! Correct on every toolchain with num_threads(1) and with fixed-size
! arrays (integer :: a(n), b(n)). With target nowait: AFAR bad= 32 at
! HSA_XNACK=0; CCE 19 intermittently bad= 32 or aborts (hipFree returned
! hipErrorInvalidValue), CCE 21 aborts at HSA_XNACK=1 (Present_table::
! release failed).
! AFAR: amdflang -fopenmp --offload-arch=gfx942 -O2
! CCE: ftn -fopenmp -O2 (module load cce/<v> craype-accel-amd-gfx942 rocm)
! Run: OMP_TARGET_OFFLOAD=mandatory HSA_XNACK=0 ./a.out
program p
implicit none
integer, parameter :: n = 64
integer, allocatable :: a(:), b(:)
integer :: i, c, lo, hi, bad
allocate(a(n), b(n))
do i = 1, n
a(i) = i
b(i) = 0
end do
!$omp parallel do num_threads(2) private(lo, hi)
do c = 1, 2
lo = (c - 1)*32 + 1
hi = c*32
!$omp target map(to: a(lo:hi)) map(from: b(lo:hi))
do i = lo, hi
b(i) = 2*a(i) + 1
end do
!$omp end target
end do
!$omp end parallel do
bad = 0
do i = 1, n
if (b(i) /= 2*i + 1) bad = bad + 1
end do
print *, 'bad=', bad
end program p
Something to add? Dispute its legality · Report it fixed in a newer release · Report a mistake on this page
Expected and observed
Expected
bad= 0
Observed where it behaves incorrectly
bad= 32
What the specification requires
The specification text this program relies on and the compiler is expected to support.
OpenMP 5.2 §5.8.1 p.148
References to structure elements or array elements are treated as references to the structure or array, respectively, for the purposes of determining implicit data-mapping or data-sharing attributes
OpenMP 5.2 §5.8.3 p.154
If a list item with an implicit data-mapping attribute does not have any corresponding storage in the device data environment prior to a task encountering the construct associated with the map clause, and one or more contiguous parts of the original storage are either list items or base pointers to list items that are explicitly mapped on the construct, only those parts of the original storage will have corresponding storage in the device data environment as a result of the map clauses on the construct.
p.154 p.154
If a single contiguous part of the original storage of a list item with an implicit data-mapping attribute has corresponding storage in the device data environment prior to a task encountering the construct on which the map clause appears, only that part of the original storage will have corresponding storage in the device data environment as a result of the map clause.
p.152 p.152
The storage for each other map clause list item becomes a distinct mappable storage block
p.156 p.156
If an array appears as a list item in a map clause, multiple parts of the array have corresponding storage in the device data environment prior to a task encountering the construct associated with the map clause, and the corresponding storage for those parts was created by maps from more than one earlier construct, the behavior is unspecified.
OpenMP 5.1 §2.21.7.1 p.354
If an array appears as a list item in a map clause, multiple parts of the array have corresponding storage in the device data environment prior to a task encountering the construct associated with the map clause, and the corresponding storage for those parts was created by maps from more than one earlier construct, the behavior is unspecified.
OpenMP 6.0 §7.9.6 p.287
Evidence
Same compiler, one change
- num_threads(1) instead of num_threads(2): AFAR 24.3.0: correct; CCE 19.0.0: correct; CCE 20.0.2: correct; CCE 21.0.3: correct; CCE 21.0.4: correct
Control program
program p implicit none integer, parameter :: n = 64 integer, allocatable :: a(:), b(:) integer :: i, c, lo, hi, bad allocate(a(n), b(n)) do i = 1, n a(i) = i b(i) = 0 end do !$omp parallel do num_threads(1) private(lo, hi) do c = 1, 2 lo = (c - 1)*32 + 1 hi = c*32 !$omp target map(to: a(lo:hi)) map(from: b(lo:hi)) do i = lo, hi b(i) = 2*a(i) + 1 end do !$omp end target end do !$omp end parallel do bad = 0 do i = 1, n if (b(i) /= 2*i + 1) bad = bad + 1 end do print *, 'bad=', bad end program p
Other compilers run it correctly
NVHPC 25.11, NVHPC 26.9