runtime error when using CUDA plugin

[cgleggett]

I'm building from git commit e6ce614, in a ubuntu 18.04 container on a centos7 host with an nvidia 2080 super GPU. All kernels throw exceptions upon execution:

Running on SYCL device GeForce RTX 2080 SUPER, driver version 440.33.01
terminate called after throwing an instance of 'cl::sycl::runtime_error'
what():  OpenCL API failed. OpenCL API returns: -30 (CL_INVALID_VALUE) -30 (CL_INVALID_VALUE)

gdb shows:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff6977801 in __GI_abort () at abort.c:79
#2  0x00007ffff7ad8957 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff7adeab6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff7adeaf1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff7aded24 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff71712b0 in cl::sycl::detail::context_impl::context_impl(std::vector<cl::sycl::device, std::allocator<cl::sycl::device> >, std::function<void (cl::sycl::exception_list)>, bool) () from /root/intel/git/build/lib/libsycl.so
#7  0x00007ffff720e478 in cl::sycl::context::context(std::vector<cl::sycl::device, std::allocator<cl::sycl::device> > const&, std::function<void (cl::sycl::exception_list)>, bool) () from /root/intel/git/build/lib/libsycl.so
#8  0x00007ffff720f173 in cl::sycl::context::context(cl::sycl::device const&, std::function<void (cl::sycl::exception_list)>, bool) ()
   from /root/intel/git/build/lib/libsycl.so
#9  0x00007ffff7231535 in cl::sycl::queue::queue(cl::sycl::device const&, std::function<void (cl::sycl::exception_list)> const&, cl::sycl::property_list const&) () from /root/intel/git/build/lib/libsycl.so
#10 0x00000000004041a9 in cl::sycl::queue::queue (this=0x7fffffffe150, SyclDevice=..., PropList=...)
    at /root/intel/git/build/lib/clang/11.0.0/include/CL/sycl/queue.hpp:74
#11 0x0000000000402ed5 in main (argc=1, argv=0x7fffffffe378) at ./simple-sycl-app.cpp:16

I'm using the same simple-sycl-app.cpp that @fwyzard gave in #1231

compilation was done with:

clang++ -fsycl -g -fsycl-targets=nvptx64-nvidia-cuda-sycldevice ./simple-sycl-app.cpp -o ssa

[cgleggett]

[editing comment as I had neglected to include the appropriate -fsycl-targets in the second issue]

the execution does work if the correct platform/device is chosen. What's confusing is that the same device shows up in two different platforms, that have the same sycl::info::platform::name, but different sycl::info::platform::version. Unfortunately the default gpu_selector picks up the device that has the runtime issue.

this device/platform fails:

      device:   GeForce RTX 2080 SUPER
      driver:   440.33.01
      platform: NVIDIA CUDA
      version:  OpenCL 1.2 CUDA 10.2.95

this device/platform works:

      device:   GeForce RTX 2080 SUPER
      driver:   CUDA 10.20
      platform: NVIDIA CUDA
      version:  CUDA 10.20

[Ruyk]

The NVIDIA OpenCL is a valid OpenCL device for the OpenCL backend when using the nvptx64-nvidia-opencl-sycldevice triple. The triple nvptx64-nvidia-cuda-sycldevice produces CUDA binaries that are only valid for the CUDA backend (i.e., version CUDA 10.20).
The GPU selector only finds a GPU, doesn't look for a specific backend, so the NVIDIA OpenCL comes up first.

[cgleggett]

using the target nvptx64-nvidia-opencl-sycldevice pulls in the 32 bit compatibility libraries:

/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
# include <gnu/stubs-32.h>

installing those leads down a rabbit hole of other missing things:

In file included from /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/postypes.h:40:
/usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/cwchar:146:11: error: no member named 'fwide' in the global namespace
  using ::fwide;
        ~~^
/usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/cwchar:147:11: error: no member named 'fwprintf' in the global namespace
  using ::fwprintf;
        ~~^
/usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/cwchar:148:11: error: no member named 'fwscanf' in the global namespace
  using ::fwscanf;
        ~~^
/usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/cwchar:158:11: error: no member named 'swprintf' in the global namespace
  using ::swprintf;
        ~~^

/usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/cwchar:203:11: error: no member named 'wscanf' in the global namespace; did you mean 'vwscanf'?
  using ::wscanf;
        ~~^
/usr/include/wchar.h:698:24: note: 'vwscanf' declared here
extern int __REDIRECT (vwscanf, (const wchar_t *__restrict __format,
                       ^
In file included from simple.cpp:1:
In file included from /root/intel/git/build/lib/clang/11.0.0/include/CL/sycl.hpp:11:
In file included from /root/intel/git/build/lib/clang/11.0.0/include/CL/sycl/accessor.hpp:12:
In file included from /root/intel/git/build/lib/clang/11.0.0/include/CL/sycl/atomic.hpp:13:
In file included from /root/intel/git/build/lib/clang/11.0.0/include/CL/sycl/detail/helpers.hpp:14:
In file included from /root/intel/git/build/lib/clang/11.0.0/include/CL/sycl/detail/common.hpp:19:
In file included from /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/string:43:
In file included from /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/localefwd.h:40:
/usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/c++locale.h:52:23: error: use of undeclared identifier 'uselocale'; did you mean 'setlocale'?
  extern "C" __typeof(uselocale) __uselocale;
                      ^
/usr/include/locale.h:122:14: note: 'setlocale' declared here
extern char *setlocale (int __category, const char *__locale) __THROW;
             ^

this starts looking like a conflict between the gcc8 environment with which llvm was built, and the system libraries.

[fwyzard]

The NVIDIA OpenCL is a valid OpenCL device for the OpenCL backend when using the nvptx64-nvidia-opencl-sycldevice triple. The triple nvptx64-nvidia-cuda-sycldevice produces CUDA binaries that are only valid for the CUDA backend (i.e., version CUDA 10.20).

I'm confused ...

I tried

$ /data/user/fwyzard/sycl/build/bin/clang++ -O2 -g -Wno-unknown-cuda-version -fsycl -fsycl-targets=nvptx64-nvidia-cuda-sycldevice -Xsycl-target-backend=nvptx64-nvidia-cuda-sycldevice '--cuda-gpu-arch=sm_35' --cuda-path=/usr/local/cuda  simple-sycl-app.cpp -o simple-sycl-app-cuda

and

$ /data/user/fwyzard/sycl/build/bin/clang++ -O2 -g -Wno-unknown-cuda-version -fsycl -fsycl-targets=nvptx64-nvidia-opencl-sycldevice -Xsycl-target-backend=nvptx64-nvidia-opencl-sycldevice '--cuda-gpu-arch=sm_35' --cuda-path=/usr/local/cuda  simple-sycl-app.cpp -o simple-sycl-app-opencl

and the two binaries that I get are virtually identical:

$ objdump -s simple-sycl-app-cuda > simple-sycl-app-cuda.dump
$ objdump -s simple-sycl-app-opencl > simple-sycl-app-opencl.dump
$ diff -u simple-sycl-app-cuda.dump simple-sycl-app-opencl.dump
--- simple-sycl-app-cuda.dump   2020-03-05 17:23:38.147930897 +0100
+++ simple-sycl-app-opencl.dump 2020-03-05 17:23:45.277931237 +0100
@@ -1,5 +1,5 @@
 
-simple-sycl-app-cuda:     file format elf64-x86-64
+simple-sycl-app-opencl:     file format elf64-x86-64
 
 Contents of section .interp:
  400238 2f6c6962 36342f6c 642d6c69 6e75782d  /lib64/ld-linux-
@@ -18778,7 +18778,7 @@
  0250 6e636c75 64652f43 4c2f7379 636c2f64  nclude/CL/sycl/d
  0260 65746169 6c2f6b65 726e656c 5f646573  etail/kernel_des
  0270 632e6870 70000400 0073696d 706c652d  c.hpp....simple-
- 0280 7379636c 2d617070 2d643461 6461352e  sycl-app-d4ada5.
+ 0280 7379636c 2d617070 2d633637 3235372e  sycl-app-c67257.
  0290 68000500 00636f6e 63757272 656e6365  h....concurrence
  02a0 2e680006 00007479 7065732e 68000700  .h....types.h...
  02b0 00737464 696e742d 75696e74 6e2e6800  .stdint-uintn.h.

[fwyzard]

using the target nvptx64-nvidia-opencl-sycldevice pulls in the 32 bit compatibility libraries:

/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
# include <gnu/stubs-32.h>

I got the same error when I forgot the -sycldevice in the target, using only -fsycl-targets=nvptx64-nvidia-opencl:

$ /data/user/fwyzard/sycl/build/bin/clang++ -O2 -g -Wno-unknown-cuda-version -fsycl -fsycl-targets=nvptx64-nvidia-opencl -Xsycl-target-backend=nvptx64-nvidia-opencl '--cuda-gpu-arch=sm_35' --cuda-path=/usr/local/cuda  simple-sycl-app.cpp -o simple-sycl-app-opencl
In file included from simple-sycl-app.cpp:1:
In file included from /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/cstdio:41:
In file included from /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/x86_64-redhat-linux/bits/c++config.h:512:
In file included from /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/x86_64-redhat-linux/bits/os_defines.h:39:
In file included from /usr/include/features.h:452:
/usr/include/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
# include <gnu/stubs-32.h>
          ^~~~~~~~~~~~~~~~
1 error generated.

[Ruyk]

NVIDIA OpenCL is now not listed as a valid DPC++ device, introduced by #1689. I think that should fix the problems, feel free to open a new ticket if something else is missing.