Observed issues › afar-xnack1-atomic-min-nested-parallel
amdflang at HSA_XNACK=1: atomic min in a parallel do nested in target teams distribute with an allocatable array faults (write to a read-only page)
Legality: independently confirmed. 2 independent readings find the program legal and the result required. How this is checked
Affected variants
| Compiler | MI300A gfx942 unified memory (XNACK 1) | MI300A gfx942 copy (XNACK 0) | MI250X gfx90a copy (XNACK 0) | MI300X gfx942 copy (XNACK 0) | MI350X gfx950 copy (XNACK 0) | A100 sm_80 separate memory | A100 sm_80 managed memory |
|---|---|---|---|---|---|---|---|
| AFAR 24.3.0 | behaves incorrectly | correct | unknown not run | correct | correct | n/a | n/a |
| CCE 19.0.0 | correct | correct | unknown not run | unknown not run | unknown not run | n/a | n/a |
| CCE 20.0.2 | correct | correct | unknown not run | unknown not run | unknown not run | n/a | n/a |
| CCE 21.0.3 | correct | correct | unknown not run | unknown not run | unknown not run | n/a | n/a |
| CCE 21.0.4 | correct | correct | unknown not run | unknown not run | unknown not run | n/a | n/a |
| NVHPC 25.11 | n/a | n/a | n/a | n/a | n/a | correct | correct |
| NVHPC 26.9 | n/a | n/a | n/a | n/a | n/a | correct | correct |
| GCC 16.2.0 | correct | correct | correct | correct | correct | correct | n/a |
"behaves incorrectly" / "not supported": this issue, confirmed in up to four environments (another build directory, another node, a repeat). "fails another way": a different failure, not counted as this issue. "unknown": not run there, or the runs disagree.
Messages
| Result | Compiler and GPU | Message |
|---|---|---|
| behaves incorrectly | AFAR 24.3.0, MI300A unified | showDisplay only launched kernel: Kernel 'omp target in _QQmain @ 9 (__omp_offloading_14_1c6acd6d__QQmain_l9)' OFFLOAD ERROR: memory access fault by GPU 6 (agent 0x5555556d35b0) at virtual address 0x555500000000. Reasons: Write access to a read-only page Use 'OFFLOAD_TRACK_ALLOCATION_TRACES=true' to track device allocations |
Reproduce
AFAR 24.3.0, MI300A unified
$AFAR/bin/amdflang -fopenmp --offload-arch=gfx942 -O2 repro.f90 -o repro HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro
CCE 19.0.0, MI300A unified (runs correctly)
module load cce/19.0.0 craype-accel-amd-gfx942 rocm/6.4.3 module unload cray-mpich ftn -fopenmp -O2 repro.f90 -o repro HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro
Program (46 lines)
! AFAR (amdflang 24.0.0git, drop 24.3.0), MI300A gfx942, -O2:
! an `atomic` min update of a mapped scalar inside a `parallel do` nested in
! `target teams distribute`, with an allocatable array written in the same
! loop, faults at HSA_XNACK=1 ("memory access fault ... Write access to a
! read-only page"); at HSA_XNACK=0 it prints r=0 s=0 (correct).
! Needed: an array written in the loop, the inner parallel do (without it: correct), and an
! atomic update whose right-hand side depends on i. An atomic add with a constant right-hand
! side is correct; r = r + mod(i, n) faults like the atomic min. In this combined form the
! allocatable o1 faults and an explicit-shape array is correct; with separate directives the
! explicit-shape array faults too (18-program matrix: data/reports/fuzz5-oop-afar.md, cluster C).
! map(always) or an explicit map of o1 do not matter.
! Hand-reduced from data/findings/fuzz/b2-183-afar.f90.
! amdflang -fopenmp --offload-arch=gfx942 -O2 afar-xnack1-atomic-min-nested-parallel.f90
! HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./a.out
program p
implicit none
integer, parameter :: n = 24, m = 2
integer, allocatable :: o1(:,:)
integer :: i, j, r
allocate(o1(n, m))
o1 = 0
r = 7
!$omp target teams distribute map(tofrom: r)
do i = 1, n
!$omp parallel do
do j = 1, m
o1(i, j) = 0
!$omp atomic
r = min(r, mod(i, n))
end do
!$omp end parallel do
end do
!$omp end target teams distribute
print '(a,i0,a,i0)', 'r=', r, ' s=', sum(o1)
end program p
! spec (requirement): In Fortran an atomic update may have the form x = intrinsic-procedure-name(x, expr-list) where
! the intrinsic is MAX or MIN.
! "x = intrinsic-procedure-name (x, expr-list)" (5.2 §4.3.1.3 p.81 l.14)
! spec (requirement): The atomic construct makes the access to the storage location atomic, so concurrent updates by
! the threads of nested parallel regions produce no indeterminate value.
! "The atomic construct ensures that a specific storage location is accessed atomically so that possible simultaneous
! reads and writes by multiple threads do not result in indeterminate values." (5.2 §15.8.4 p.311 l.29-30)
! spec (this program): The atomic min update of the mapped scalar r runs in a parallel do nested in target teams
! distribute and must give r = 0; the spec has no text on memory faults with unified-memory settings (HSA_XNACK=1 is
! a vendor runtime setting), so the cited text states only that the update must complete correctly, and the write
! access fault to a read-only page is a deviation from it.
Something to add? Dispute its legality · Report it fixed in a newer release · Report a mistake on this page
Expected and observed
Expected
r=0 s=0
Observed where it behaves incorrectly
Display only launched kernel: Kernel 'omp target in _QQmain @ 9 (__omp_offloading_14_1c6acd6d__QQmain_l9)' OFFLOAD ERROR: memory access fault by GPU 6 (agent 0x5555556d35b0) at virtual address 0x555500000000. Reasons: Write access to a read-only page Use 'OFFLOAD_TRACK_ALLOCATION_TRACES=true' to track device allocations
What the specification requires
The specification text this program relies on and the compiler is expected to support.
OpenMP 5.2 §4.3.1.3 p.81 l.14
x = intrinsic-procedure-name (x, expr-list)
OpenMP 5.2 §15.8.4 p.311 l.29-30
The atomic construct ensures that a specific storage location is accessed atomically so that possible simultaneous reads and writes by multiple threads do not result in indeterminate values.
Evidence
Same compiler, one change
- o1 an explicit-shape array instead of an allocatable: AFAR 24.3.0: correct; CCE 19.0.0: correct; CCE 20.0.2: correct; CCE 21.0.3: correct; CCE 21.0.4: correct
Control program
program p implicit none integer, parameter :: n = 24, m = 2 integer :: o1(n, m) integer :: i, j, r o1 = 0 r = 7 !$omp target teams distribute map(tofrom: r) do i = 1, n !$omp parallel do do j = 1, m o1(i, j) = 0 !$omp atomic r = min(r, mod(i, n)) end do !$omp end parallel do end do !$omp end target teams distribute print '(a,i0,a,i0)', 'r=', r, ' s=', sum(o1) end program p
Other compilers run it correctly
CCE 19.0.0, CCE 20.0.2, CCE 21.0.3, CCE 21.0.4, NVHPC 25.11, NVHPC 26.9, GCC 16.2.0