Observed issues › catalog/afar-flang-dist-schedule-chunk
Appears to behave incorrectlyWrong answer · Fortran · OpenMP · Open
amdflang: dist_schedule(static, 4) is not applied: 64 iterations over 4 teams go to teams 0 and 1 in two blocks of 32
Legality: independently confirmed. 2 independent readings find the program legal and the result required. How this is checked
Affected variants
| Compiler | MI300A gfx942 unified memory (XNACK 1) | MI300A gfx942 copy (XNACK 0) | MI250X gfx90a copy (XNACK 0) | MI300X gfx942 copy (XNACK 0) | MI350X gfx950 copy (XNACK 0) | A100 sm_80 separate memory | A100 sm_80 managed memory |
|---|---|---|---|---|---|---|---|
| AFAR 24.3.0 | behaves incorrectly | behaves incorrectly | behaves incorrectly | behaves incorrectly | behaves incorrectly | n/a | n/a |
| CCE 19.0.0 | correct | correct | correct | unknown not run | unknown not run | n/a | n/a |
| CCE 20.0.2 | correct | correct | correct | unknown not run | unknown not run | n/a | n/a |
| CCE 21.0.3 | correct | correct | correct | unknown not run | unknown not run | n/a | n/a |
| CCE 21.0.4 | correct | correct | correct | unknown not run | unknown not run | n/a | n/a |
| NVHPC 25.11 | n/a | n/a | n/a | n/a | n/a | fails another way | fails another way |
| NVHPC 26.9 | n/a | n/a | n/a | n/a | n/a | fails another way | fails another way |
| GCC 16.2.0 | correct | correct | correct | correct | correct | correct | n/a |
"behaves incorrectly" / "not supported": this issue, confirmed in up to four environments (another build directory, another node, a repeat). "fails another way": a different failure, not counted as this issue. "unknown": not run there, or the runs disagree.
Messages
| Result | Compiler and GPU | Message |
|---|---|---|
| behaves incorrectly | AFAR 24.3.0, MI300A unified AFAR 24.3.0, MI300A copy AFAR 24.3.0, MI250X AFAR 24.3.0, MI300X AFAR 24.3.0, MI350X | showRESULT ran_on_host=0 check=fail |
| fails another way | NVHPC 25.11, A100 separate NVHPC 25.11, A100 managed NVHPC 26.9, A100 separate NVHPC 26.9, A100 managed | showFailing in Thread:1 Accelerator Fatal Error: call to cuMemcpyDtoHAsync returned error 700 (CUDA_ERROR_ILLEGAL_ADDRESS): Illegal address during kernel execution File: <dir>/afar-flang-dist-schedule-chunk.f90 Function: gen_prog:1 Line: 29 |
Reproduce
AFAR 24.3.0, MI300A unified
$AFAR/bin/amdflang -fopenmp --offload-arch=gfx942 -O2 repro.f90 -o repro HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro
CCE 19.0.0, MI300A unified (runs correctly)
module load cce/19.0.0 craype-accel-amd-gfx942 rocm/6.4.3 module unload cray-mpich ftn -fopenmp -O2 repro.f90 -o repro HSA_XNACK=1 OMP_TARGET_OFFLOAD=mandatory ./repro
Program (73 lines)
! # Draft finding: omp52-clause-dist_schedule-chunk_size on afar
!
! - compiler: afar
! - toolchain: afar; arch: gfx942; machine: tuolumne; memory mode: unified; opt: -O2
! - program: catalog omp52-clause-dist_schedule-chunk_size (f90, version 52, sha256 3628c7a53985f9c2)
! - expected: check=pass (trusted; derivation + model + probe)
! - actual: wrong_answer check=fail (check_fail: ran_on_host=0 check=fail)
! - verdict: compiler_bug by rule R20-runtime-bug: a legal program fails at run time, deterministically, against an expected value
! from two or more independent sources with an executed witness, and another toolchain, mode, arch or the elision gives the
! expected value
! - reason: expected trusted check=pass (derivation + model + probe): stored main verdict: derivation + model + probe agree;
! witnesses: afar/gfx942/copy (port:omp52-c-clause-dist_schedule-chunk_size), afar/gfx942/unified
! (port:omp52-c-clause-dist_schedule-chunk_size), cce19/gfx942/copy, cce19/gfx942/unified, cce20/gfx942/copy,
! cce20/gfx942/unified, cce21/gfx942/copy, cce21/gfx942/unified
! - citations: RUL-DERIVED-NEEDS-EXECUTION (rulings.md); RUL-WITNESS-PER-MODE (rulings.md);
! RUL-SAME-COMPILER-RUNS (rulings.md)
! - legality: legal (legal by construction: offload.engine.gen / c_gen / acc_gen check every program against the catalog rules
! and the
! offload scope as it is built (no compiler))
! - witnesses: afar/gfx942/copy (port:omp52-c-clause-dist_schedule-chunk_size), afar/gfx942/unified
! (port:omp52-c-clause-dist_schedule-chunk_size), cce19/gfx942/copy, cce19/gfx942/unified, cce20/gfx942/copy,
! cce20/gfx942/unified, cce21/gfx942/copy, cce21/gfx942/unified
! - withdrawn: none
!
! Generated by compliance.gen (OpenMP 5.2, free-form Fortran). Do not edit.
! id: omp52-clause-dist_schedule-chunk_size
! primary atom: omp.clause.dist_schedule.chunk_size
! legal: yes
! observable: yes
! ran_on_host: inside the target region of interest
! evidence: construct
! needs: none
! std: f2008
! derivation (check=pass): dist_schedule(static, 4): chunks of 4 iterations assigned to the initial teams round-robin in team-number
! derivation: order (5.2 §11.6.1 p.256), so iteration i runs on team mod((i - 1)/4, nt), which dok tests; nt >= 1 and b = ref; the
! derivation: check accepts
program gen_prog
use omp_lib, only: omp_get_num_teams, omp_get_team_num, omp_is_initial_device
implicit none
integer, parameter :: n = 64
integer, allocatable, target :: a(:), b(:)
integer :: ref(n)
integer :: i, host_flag
logical :: ok
integer :: team(n), nt
logical :: dok
allocate(a(n), b(n))
do i = 1, n
a(i) = i
b(i) = 0
ref(i) = 2*i + 1
end do
host_flag = 1
team = -1
nt = -1
!$omp target teams distribute parallel do map(to: a) map(from: b) map(from: team, nt) &
!$omp& map(tofrom: host_flag) default(none) shared(a, b, team, nt, host_flag) num_teams(4) &
!$omp& dist_schedule(static, 4)
do i = 1, n
b(i) = 2*a(i) + 1
team(i) = omp_get_team_num()
if (i == 1) nt = omp_get_num_teams()
if (i == 1) host_flag = merge(1, 0, omp_is_initial_device())
end do
!$omp end target teams distribute parallel do
dok = .true.
do i = 1, n
dok = dok .and. team(i) == mod((i - 1)/4, nt)
end do
ok = all(b == ref) .and. (nt >= 1) .and. dok
print '(a,i0,a,a)', 'RESULT ran_on_host=', host_flag, ' check=', merge('pass', 'fail', ok)
deallocate(a, b)
end program gen_prog
Something to add? Dispute its legality · Report it fixed in a newer release · Report a mistake on this page
Expected and observed
Expected
check=pass
Observed where it behaves incorrectly
RESULT ran_on_host=0 check=fail
What the specification requires
The specification text this program relies on and the compiler is expected to support.
OpenMP 5.2 §11.6.1 p.256
Evidence
Same compiler, one change
- num_teams(1) instead of num_teams(4) (every chunk on team 0): AFAR 24.3.0: correct; CCE 19.0.0: correct; CCE 20.0.2: correct; CCE 21.0.3: correct; CCE 21.0.4: correct
Control program
program gen_prog use omp_lib, only: omp_get_num_teams, omp_get_team_num, omp_is_initial_device implicit none integer, parameter :: n = 64 integer, allocatable, target :: a(:), b(:) integer :: ref(n) integer :: i, host_flag logical :: ok integer :: team(n), nt logical :: dok allocate(a(n), b(n)) do i = 1, n a(i) = i b(i) = 0 ref(i) = 2*i + 1 end do host_flag = 1 team = -1 nt = -1 !$omp target teams distribute parallel do map(to: a) map(from: b) map(from: team, nt) & !$omp& map(tofrom: host_flag) default(none) shared(a, b, team, nt, host_flag) num_teams(1) & !$omp& dist_schedule(static, 4) do i = 1, n b(i) = 2*a(i) + 1 team(i) = omp_get_team_num() if (i == 1) nt = omp_get_num_teams() if (i == 1) host_flag = merge(1, 0, omp_is_initial_device()) end do !$omp end target teams distribute parallel do dok = .true. do i = 1, n dok = dok .and. team(i) == mod((i - 1)/4, nt) end do ok = all(b == ref) .and. (nt >= 1) .and. dok print '(a,i0,a,a)', 'RESULT ran_on_host=', host_flag, ' check=', merge('pass', 'fail', ok) deallocate(a, b) end program gen_prog
Other compilers run it correctly
CCE 19.0.0, CCE 20.0.2, CCE 21.0.3, CCE 21.0.4, GCC 16.2.0