OffTarget

Observed issues › fuzz/nvhpc-num-threads-reduction-identity-lost

Appears to behave incorrectlyWrong answer · Fortran · OpenMP · Open

num_threads with a * reduction gives 0

Legality: independently confirmed. 2 independent readings find the program legal and the result required. How this is checked

nvfortran -mp=gpu appears to behave incorrectly on a worksharing-loop reduction with *, max or iand in a parallel region with num_threads(2) inside a target region: the result is 0. Without num_threads, or with a + reduction, the result is right.

Affected variants

CompilerMI300A gfx942
unified memory (XNACK 1)
MI300A gfx942
copy (XNACK 0)
MI250X gfx90a
copy (XNACK 0)
MI300X gfx942
copy (XNACK 0)
MI350X gfx950
copy (XNACK 0)
A100 sm_80
separate memory
A100 sm_80
managed memory
AFAR 24.3.0correctcorrectunknown
not run
correctcorrectn/an/a
CCE 19.0.0correctcorrectunknown
not run
unknown
not run
unknown
not run
n/an/a
CCE 20.0.2correctcorrectunknown
not run
unknown
not run
unknown
not run
n/an/a
CCE 21.0.3correctcorrectunknown
not run
unknown
not run
unknown
not run
n/an/a
CCE 21.0.4correctcorrectunknown
not run
unknown
not run
unknown
not run
n/an/a
NVHPC 25.11n/an/an/an/an/abehaves incorrectlybehaves incorrectly
NVHPC 26.9n/an/an/an/an/abehaves incorrectlybehaves incorrectly
GCC 16.2.0correctcorrectcorrectcorrectcorrectcorrectn/a

"behaves incorrectly" / "not supported": this issue, confirmed in up to four environments (another build directory, another node, a repeat). "fails another way": a different failure, not counted as this issue. "unknown": not run there, or the runs disagree.

Messages

ResultCompiler and GPUMessage
behaves incorrectlyNVHPC 25.11, A100 separate
NVHPC 25.11, A100 managed
NVHPC 26.9, A100 separate
NVHPC 26.9, A100 managed
show
r= 0
RESULT ran_on_host=0 check=fail

Reproduce

NVHPC 25.11, A100 separate

export LD_LIBRARY_PATH=$NVHPC_25_11/compilers/lib
nvfortran -mp=gpu -gpu=cc80 -gpu=mem:separate -O2 repro.f90 -o repro
OMP_TARGET_OFFLOAD=mandatory ./repro

NVHPC 26.9, A100 separate

export LD_LIBRARY_PATH=$NVHPC_26_9/compilers/lib
nvfortran -mp=gpu -gpu=cc80 -gpu=mem:separate -O2 repro.f90 -o repro
OMP_TARGET_OFFLOAD=mandatory ./repro

AFAR 24.3.0, MI300A unified (runs correctly)

$AFAR/bin/amdflang -fopenmp --offload-arch=gfx942 -O2 repro.f90 -o repro
HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro
Program (47 lines)
! # Finding (parity-fuzz-c1): NVHPC: a reduction(*) (also max, iand) on the worksharing loop of a parallel region with
! # num_threads(2) in a target region gives 0
!
! - toolchains and modes that fail: nvhpc (NVHPC 25.11), nvhpc269 (NVHPC 26.9); modes: copy, managed; -O2
! - build: nvfortran -mp=gpu -gpu=cc80 -gpu=mem:separate -O2
! - failure: wrong_answer (check=fail)
! - observed: `r= 0` with `check=fail`; identical signature on 4 of 4 failing cells
! - expected: compiles and prints `r= 768` with `check=pass`
! - rule: "For each list item, a private copy is created for each implicit task or SIMD lane and is initialized with the initializer
!   value of the reduction-identifier. After the end of the region, the original list item is updated with the values of the private
!   copies using the combiner associated with the reduction-identifier." (5.2 §5.5.8 p.134 l.21-24); the initializer of * is 1
!   (Table 5.2), so r is 3 times 2**8 = 768 whatever the number of threads
! - vendor: on GPU targets the parallel construct supports "The num_threads, default, private, firstprivate, and shared clauses",
!   and "The reduction clause is supported. The optional modifier is not supported." (NVIDIA HPC Compilers User's Guide 26.9, Using
!   OpenMP, items 2.6 and 2.19.5; data/index/claims/nvidia-hpc-26.9-user-guide-openmp.txt)
! - same-compiler control: controls/nvhpc-num-threads-removed-control.f90 (one change from this file; correct on nvhpc (NVHPC
!   25.11), nvhpc269 (NVHPC 26.9); rows results/parity/nvc1/tuo.nt.jsonl, results/parity/nvc1/wt.nt.jsonl)
! - other families: afar (AFAR 24.3.0), cce19 (CCE 19.0.0), cce20 (CCE 20.0.2), cce21 (CCE 21.0.3), cce21_4 (CCE 21.0.4) run the
!   file correctly (rows results/parity/nvc1/tuo.nt.jsonl)
! - variants: files next to the reduction in results/parity/nvc1/red/, rows results/parity/nvc1/wt.nt.jsonl (both releases, separate
!   and managed memory) and results/parity/nvc1/tuo.nt.jsonl: reduction(max: r) (nvhpc-num-threads-reduction-max-variant.f90) and
!   reduction(iand: r) (nvhpc-num-threads-reduction-iand-variant.f90) print `r= 0` too, as does the same loop written as target,
!   parallel num_threads(2) and do (nvhpc-num-threads-split-variant.f90); reduction(+: r) with num_threads(2)
!   (nvhpc-num-threads-reduction-sum-variant.f90) is correct. The reduction identifiers whose initializer is not 0 lose their result
! - gate and confirmer: 4 of 4 cells reproduce in all four environments (a-d; (c) on another node or GPU than (a); rows
!   results/parity/nvc1/gate_abd.nt.jsonl, results/parity/nvc1/gate_abd.nt.m2.jsonl, results/parity/nvc1/gate_c.nt.jsonl,
!   results/parity/nvc1/gate_c.nt.m2.jsonl, results/parity/nvc1/wt.nt.jsonl); campaign gate of the programs below: 2 cells, 2
!   confirmed in the four environments (a-d; (c) on another node or GPU; rows
!   results/parity/wingtip/results/parity/gate/fuzz/reruns/wingtip.abd.127721.jsonl,
!   results/parity/wingtip/results/parity/gate/fuzz/reruns/wingtip.c.210952.jsonl)
! - programs (1): par-c1-nvhpc-nvhpc269:2123
! - legality: the rules (offload.engine.rules) and AFAR's flang front end accept this file, its control and the variants above
!   (offload.engine.prerun.check on a Tuolumne compute node; results/parity/nvc1/prerun.nt.txt); AFAR 24.3.0 and CCE 19/20/21/21.0.4
!   compile and run all of them correctly with HSA_XNACK 0 and 1 (results/parity/nvc1/tuo.nt.jsonl). NVHPC is the compiler under
!   test and is not counted as a witness
program p
  implicit none
  integer :: r, i
  r = 3
  !$omp target parallel do map(tofrom: r) num_threads(2) reduction(*: r)
  do i = 1, 8
    r = r * 2
  end do
  !$omp end target parallel do
  print '(a,i0)', 'r= ', r
  print '(a,a)', 'RESULT ran_on_host=0 check=', merge('pass', 'fail', r == 768)
end program p

Source in the repository

Something to add? Dispute its legality · Report it fixed in a newer release · Report a mistake on this page

Expected and observed

Observed where it behaves incorrectly

r= 0
RESULT ran_on_host=0 check=fail

What the specification requires

The specification text this program relies on and the compiler is expected to support.

OpenMP 5.2 §5.5.8 p.134 l.21-24

For each list item, a private copy is created for each implicit task or SIMD lane and is initialized with the initializer value of the reduction-identifier. After the end of the region, the original list item is updated with the values of the private copies using the combiner associated with the reduction-identifier.

Evidence

Same compiler, one change

  • one change from the finding; correct on nvhpc, nvhpc269: not run yet
    Control program
    program p
      implicit none
      integer :: r, i
      r = 3
      !$omp target parallel do map(tofrom: r) reduction(*: r)
      do i = 1, 8
        r = r * 2
      end do
      !$omp end target parallel do
      print '(a,i0)', 'r= ', r
      print '(a,a)', 'RESULT ran_on_host=0 check=', merge('pass', 'fail', r == 768)
    end program p
    

Other compilers run it correctly

AFAR 24.3.0, CCE 19.0.0, CCE 20.0.2, CCE 21.0.3, CCE 21.0.4, GCC 16.2.0