OffTarget

Observed issues › harvest/cce-acc-declare-create-module-allocatable-xnack1-fault

Appears to behave incorrectlyCrashes or aborts at run time · Fortran · OpenACC · Open

ftn (OpenACC): a module allocatable in declare create faults at HSA_XNACK=1

Legality: independently confirmed. 2 independent readings find the program legal and the result required. How this is checked

Affected variants

CompilerMI300A gfx942
unified memory (XNACK 1)
MI300A gfx942
copy (XNACK 0)
MI250X gfx90a
copy (XNACK 0)
MI300X gfx942
copy (XNACK 0)
MI350X gfx950
copy (XNACK 0)
A100 sm_80
separate memory
A100 sm_80
managed memory
AFAR 24.3.0n/an/an/an/an/an/an/a
CCE 19.0.0behaves incorrectlycorrectunknown
not run
unknown
not run
unknown
not run
n/an/a
CCE 20.0.2behaves incorrectlycorrectunknown
not run
unknown
not run
unknown
not run
n/an/a
CCE 21.0.3behaves incorrectlycorrectunknown
not run
unknown
not run
unknown
not run
n/an/a
CCE 21.0.4behaves incorrectlycorrectunknown
not run
unknown
not run
unknown
not run
n/an/a
NVHPC 25.11n/an/an/an/an/acorrectcorrect
NVHPC 26.9n/an/an/an/an/acorrectcorrect
GCC 16.2.0behaves incorrectlybehaves incorrectlybehaves incorrectlybehaves incorrectlybehaves incorrectlybehaves incorrectlyn/a

"behaves incorrectly" / "not supported": this issue, confirmed in up to four environments (another build directory, another node, a repeat). "fails another way": a different failure, not counted as this issue. "unknown": not run there, or the runs disagree.

Messages

ResultCompiler and GPUMessage
behaves incorrectlyCCE 19.0.0, MI300A unified
show
Memory access fault by GPU node-7 (Agent handle: 0x4d5c50) on address (nil). Reason: Write access to a read-only page.
behaves incorrectlyCCE 20.0.2, MI300A unified
show
Memory access fault by GPU node-7 (Agent handle: 0x4d6eb0) on address 0x6000. Reason: Write access to a read-only page.
behaves incorrectlyCCE 21.0.3, MI300A unified
CCE 21.0.4, MI300A unified
show
ACC: libcrayacc/acc_hw_amd.c:1927 CRAY_ACC_ERROR -  hipStreamSynchronize returned hipErrorIllegalAddress: 'an illegal memory access was encountered' from site-harvest--cce-acc-declare-create-module-al
ACC: libcrayacc/acc_hw_amd.c:1476 CRAY_ACC_ERROR -  hipDeviceSynchronize returned hipErrorIllegalAddress: 'an illegal memory access was encountered'
behaves incorrectlyGCC 16.2.0, MI300A unified
GCC 16.2.0, MI300A copy
show
libgomp: [0x8dfdd0,32768] is not mapped
behaves incorrectlyGCC 16.2.0, MI250X
show
libgomp: [0xa84a70,32768] is not mapped
behaves incorrectlyGCC 16.2.0, MI300X
show
libgomp: [0x1098e80,32768] is not mapped
behaves incorrectlyGCC 16.2.0, MI350X
show
libgomp: [0x20d8ad0,32768] is not mapped
behaves incorrectlyGCC 16.2.0, A100 separate
show
libgomp: [0x12e37f40,32768] is not mapped

Reproduce

CCE 19.0.0, MI300A unified

module load cce/19.0.0 craype-accel-amd-gfx942 rocm/6.4.3
module unload cray-mpich
ftn -h acc -O2 repro.f90 -o repro
HSA_XNACK=1 ./repro

CCE 20.0.2, MI300A unified

module load cce/20.0.2 craype-accel-amd-gfx942 rocm/6.4.3
module unload cray-mpich
ftn -h acc -O2 repro.f90 -o repro
HSA_XNACK=1 ./repro

CCE 21.0.3, MI300A unified

module load cce/21.0.3 craype-accel-amd-gfx942 rocm/7.2.0
module unload cray-mpich
ftn -h acc -O2 repro.f90 -o repro
HSA_XNACK=1 ./repro

CCE 21.0.4, MI300A unified

module load cce/21.0.4 craype-accel-amd-gfx942 rocm/7.2.0
module unload cray-mpich
ftn -h acc -O2 repro.f90 -o repro
HSA_XNACK=1 ./repro

GCC 16.2.0, MI300A unified

export LD_LIBRARY_PATH=$GCC_16_2/lib64:/opt/rocm-7.2.1/lib
$GCC_16_2/bin/gfortran -fopenacc -foffload=amdgcn-amdhsa -foffload-options=amdgcn-amdhsa=-march=gfx942 -O2 repro.f90 -o repro
HSA_XNACK=1 ./repro

GCC 16.2.0, A100 separate

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=$GCC_16_2/lib64
$GCC_16_2/bin/gfortran -fopenacc -foffload=nvptx-none -foffload-options=nvptx-none=-march=sm_80 -foffload-options=nvptx-none=-latomic -O2 repro.f90 -o repro
./repro

NVHPC 25.11, A100 separate (runs correctly)

export LD_LIBRARY_PATH=$NVHPC_25_11/compilers/lib
nvfortran -acc=gpu -gpu=cc80 -gpu=mem:separate -O2 repro.f90 -o repro
./repro
Program (55 lines)
! # Finding (known-issues harvest): CCE faults on a module allocatable in
!   declare create at HSA_XNACK=1 (MI300A)
!
! - source: MFC's compiler-ticket list (PE-53844, "CCE >17 crashes on host_data use_device for
!   module variables"), reconstructed as data/bench/harvest-known hk-008 (a declare create module
!   allocatable, host_data use_device, a deviceptr routine); the fault is at the first compute
!   construct, before host_data, so this file drops host_data
!   (data/reports/harvest-known/v008-declare-create-alloc.f90)
! - compilers: CCE 19.0.0, 20.0.2, 21.0.3, ftn -h acc -O2, MI300A at HSA_XNACK=1 (THP always)
! - expected: bad = 0, RESULT ... check=pass. For an allocatable in declare create, "for an
!   allocate statement for var ... memory will be allocated in both local memory as well as in the
!   current device memory" on a non-shared memory device (OpenACC 3.4 §2.13.2 p.83 l.3053-3057); on
!   a shared memory device the host memory is the device's; either way buf is present in the
!   parallel loop and the update copies (or leaves) the values
! - actual: CCE 19 and 20: "Memory access fault by GPU node-4 ... on address 0x6000. Reason: Write
!   access to a read-only page." (SIGABRT); CCE 21: "hipStreamSynchronize returned
!   hipErrorIllegalAddress ... from v008-declare-create-alloc.f90:19" (the parallel loop), exit 1
! - control (same compiler): HSA_XNACK=0 passes; at HSA_XNACK=1 a local allocatable made present
!   with enter data create passes (c008-local-enter-data.f90), and a fixed-size module array in
!   declare create passes (c008-fixed-module.f90), both in data/reports/harvest-known/
! - other families: NVHPC 25.11 and 26.9 (OpenACC; separate and managed memory) pass; AFAR has no
!   OpenACC
! - legality: the pre-run check (rules and the AFAR front end with -fopenacc) accepts it; present
!   requires the data present, which declare create and the allocate statement provide
! - also: hk-007 (MFC ticket OLCFHELP-19298 / HPECOE-71) and hk-013 (thornado commit 2253976852,
!   "another CCE bug") hold a module allocatable in declare create too, and fail the same way on CCE
!   19/20/21 at HSA_XNACK=1 only (19 and 20 fault, 21 hipErrorIllegalAddress; another node)
! - evidence: results/harvest-known/main (hk-007, hk-008, hk-013, CCE 19/20/21 at HSA_XNACK=1),
!   results/harvest-known/probes.tuolumne.jsonl, probes.wingtip.jsonl
module v008_m
  implicit none
  real(8), allocatable :: buf(:)
  !$acc declare create(buf)
end module v008_m

program v008
  use openacc
  use v008_m
  implicit none
  integer, parameter :: n = 4096
  integer :: i, bad, hk_host
  allocate(buf(n))
  !$acc parallel loop present(buf)
  do i = 1, n
    buf(i) = real(i, 8)
  end do
  !$acc update host(buf)
  bad = count(buf /= [(real(i, 8), i = 1, n)])
  hk_host = 1
  !$acc parallel num_gangs(1) copy(hk_host)
  if (.not. acc_on_device(acc_device_host)) hk_host = 0
  !$acc end parallel
  print '(a,i0)', 'bad = ', bad
  print '(a,i0,a,a)', 'RESULT ran_on_host=', hk_host, ' check=', merge('pass', 'fail', bad == 0)
end program v008

Source in the repository

Something to add? Dispute its legality · Report it fixed in a newer release · Report a mistake on this page

Expected and observed

Expected

check=pass

Observed where it behaves incorrectly

Memory access fault by GPU node-7 (Agent handle: 0x4d5c50) on address (nil). Reason: Write access to a read-only page.

What the specification requires

The specification text this program relies on and the compiler is expected to support.

OpenACC 3.4 §2.13.2 p.83 l.3053-3057

Evidence

Other compilers run it correctly

NVHPC 25.11, NVHPC 26.9