CUDA back-end fails with OpenCL header problems

[jeffhammond]

I am using 69825ee:

$ git pull
Already up-to-date.
$ git branch
* sycl
$ git log -n1
commit 69825ee6cee5f0e2827ddd4b5b50fd0a5ba01cdd
Author: vladimirlaz <vladimir.lazarev@intel.com>
Date:   Thu Oct 8 21:56:39 2020 +0300
...

I build like this:

$ git clean -dfx
$ /swtools/python/python-3.5.1/bin/python3 ./buildbot/configure.py --cuda  && /swtools/python/python-3.5.1/bin/python3 ./buildbot/compile.py 
args:Namespace(arm=False, base_branch=None, branch=None, build_number=None, build_type='Release', builder_dir=None, cmake_gen='Ninja', cmake_opt=None, cuda=True, docs=False, l0_headers=None, l0_loader=None, no_assertions=False, no_werror=False, obj_dir=None, pr_number=None, shared_libs=False, src_dir=None, system_ocl=False)
[Cmake Command]: cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=X86;NVPTX -DLLVM_EXTERNAL_PROJECTS=sycl;llvm-spirv;opencl-aot;xpti;libdevice -DLLVM_EXTERNAL_SYCL_SOURCE_DIR=~/ISYCL/llvm/sycl -DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=~/ISYCL/llvm/llvm-spirv -DLLVM_EXTERNAL_XPTI_SOURCE_DIR=~/ISYCL/llvm/xpti -DLLVM_EXTERNAL_LIBDEVICE_SOURCE_DIR=~/ISYCL/llvm/libdevice -DLLVM_ENABLE_PROJECTS=clang;llvm-spirv;sycl;opencl-aot;xpti;libdevice;libclc -DLIBCLC_TARGETS_TO_BUILD=nvptx64--;nvptx64--nvidiacl -DSYCL_BUILD_PI_CUDA=ON -DLLVM_BUILD_TOOLS=ON -DSYCL_ENABLE_WERROR=ON -DCMAKE_INSTALL_PREFIX=~/ISYCL/llvm/build/install -DSYCL_INCLUDE_TESTS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DLLVM_ENABLE_SPHINX=OFF -DBUILD_SHARED_LIBS=OFF -DSYCL_ENABLE_XPTI_TRACING=ON ~/ISYCL/llvm/llvm
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /swtools/gcc/7.4.0/bin/gcc
-- Check for working C compiler: /swtools/gcc/7.4.0/bin/gcc
-- Check for working C compiler: /swtools/gcc/7.4.0/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /swtools/gcc/7.4.0/bin/g++
-- Check for working CXX compiler: /swtools/gcc/7.4.0/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
...
LLVM libdir: ~/ISYCL/llvm/build/./lib
LLVM bindir: ~/ISYCL/llvm/build/./bin
LLVM cxx flags: -fno-rtti;-fno-exceptions

clang: ~/ISYCL/llvm/build/./bin/clang
llvm-as: ~/ISYCL/llvm/build/./bin/llvm-as
llvm-link: ~/ISYCL/llvm/build/./bin/llvm-link
opt: ~/ISYCL/llvm/build/./bin/opt
llvm-spirv: LLVM_SPIRV-NOTFOUND

BUILDING nvptx64--
    DEVICE: none (  )
BUILDING nvptx64--nvidiacl
    DEVICE: none (  )
 Testing : nvptx64--nvidiacl
-- OpenCL_INCLUDE_DIRS is missing. Will try to download OpenCL headers from github.com
-- OpenCL_LIBRARIES is missing. Will try to download OpenCL ICD Loader from github.com
-- Including the PI API CUDA backend.
...

The tool fails like this:

/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice saxpy.cc -o saxpy.x
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
In file included from saxpy.cc:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:11:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/cl_ext_intel.h:431:9: error: unknown type name 'cl_properties'
typedef cl_properties cl_mem_properties_intel;
        ^
In file included from saxpy.cc:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:121:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/exception.hpp:15:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:228:7: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES'
      CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES,
      ^
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:229:45: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_HOST_PROPERTIES'
  PI_DEVICE_INFO_QUEUE_ON_HOST_PROPERTIES = CL_DEVICE_QUEUE_ON_HOST_PROPERTIES,
                                            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
4 errors generated.
make: *** [saxpy.x] Error 1

This is the source code:

#include <iostream>

#include <cmath>
#include <cfloat>

#include "CL/sycl.hpp"

namespace sycl = cl::sycl;

const float xval(1);
const float yval(2);
const float zval(2);
const float aval(3);

const float correct = (zval + aval * xval + yval);

int main(int argc, char * argv[])
{
    if (argc < 2) {
        std::cerr << "Usage: saxpy.x <vector length>" << std::endl;
        return argc;
    }

    size_t length = std::atoi(argv[1]);
    std::cout << "SAXPY with " << length << " elements" << std::endl;

    // host data
    std::vector<float> h_X(length,xval);
    std::vector<float> h_Y(length,yval);
    std::vector<float> h_Z(length,zval);

    sycl::queue q(sycl::default_selector{});

    try {

        const float A(aval);

        sycl::buffer<float,1> d_X { h_X.data(), sycl::range<1>(h_X.size()) };
        sycl::buffer<float,1> d_Y { h_Y.data(), sycl::range<1>(h_Y.size()) };
        sycl::buffer<float,1> d_Z { h_Z.data(), sycl::range<1>(h_Z.size()) };

        q.submit([&](sycl::handler& h) {

#if 0
            auto X = d_X.get_access<sycl::access::mode::read>(h);
            auto Y = d_Y.get_access<sycl::access::mode::read>(h);
            auto Z = d_Z.get_access<sycl::access::mode::read_write>(h);
#else
            sycl::accessor X(d_X,h,sycl::read_only);
            sycl::accessor Y(d_Y,h,sycl::read_only);
            sycl::accessor Z(d_Z,h,sycl::read_write);
#endif

            h.parallel_for<class axpy>( sycl::range<1>{length}, [=] (sycl::id<1> i) {
                Z[i] += A * X[i] + Y[i];
            });
        });
        q.wait();
    }
    catch (sycl::exception & e) {
        std::cout << e.what() << std::endl;
        return 1;
    }

    // check for correctness
    size_t errors(0);
    for (size_t i=0; i<length; ++i) {
        if ( std::abs(h_Z[i] - correct) > FLT_MIN) {
            ++errors;
        }
    }
    if (errors) {
        std::cerr << "There were " << errors << " errors!" << std::endl;
        for (size_t i=0; i<length; ++i) {
            std::cout << i << "," << h_Z[i] << "\n";
        }
        return 1;
    }
    std::cout << "Program completed without error." << std::endl;

    return 0;
}

[bader]

I can't reproduce this problem. Here is what I do:

export CUDADIR=/usr/local/cuda-10.1
export PATH=$PATH:$CUDADIR/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDADIR/lib64
git clone https://github.com/intel/llvm -b sycl
cd llvm/
python3 ./buildbot/configure.py --cuda
python3 ./buildbot/compile.py
vim /tmp/test.cpp # add reproducer source
./build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice /tmp/test.cpp -o /tmp/test.o

[bader]

@jeffhammond, were you able to compile the reproducer using these instructions?

I can't reproduce this problem. Here is what I do:

export CUDADIR=/usr/local/cuda-10.1
export PATH=$PATH:$CUDADIR/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDADIR/lib64
git clone https://github.com/intel/llvm -b sycl
cd llvm/
python3 ./buildbot/configure.py --cuda
python3 ./buildbot/compile.py
vim /tmp/test.cpp # add reproducer source
./build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice /tmp/test.cpp -o /tmp/test.o

[jeffhammond]

Clean build from the latest git and it's still broken. I am using the buildbot recipe just like you.

[jrhammon@pcl-tiergarten src]$ /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang++  -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice test.cpp 
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
In file included from test.cpp:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:11:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/cl_ext_intel.h:431:9: error: unknown type name 'cl_properties'
typedef cl_properties cl_mem_properties_intel;
        ^
In file included from test.cpp:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:121:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/exception.hpp:15:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:228:7: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES'
      CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES,
      ^
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:229:45: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_HOST_PROPERTIES'
  PI_DEVICE_INFO_QUEUE_ON_HOST_PROPERTIES = CL_DEVICE_QUEUE_ON_HOST_PROPERTIES,
                                            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
4 errors generated.

[jeffhammond]

What I don't understand is why there is any reference to cl_ext_intel.h when I build for CUDA. Why does the compiler need to know anything about Intel OpenCL to do a non-OpenCL CUDA build?

[lfmeadow]

Look at this:
-- OpenCL_INCLUDE_DIRS is missing. Will try to download OpenCL headers from github.com
-- OpenCL_LIBRARIES is missing. Will try to download OpenCL ICD Loader from github.com
I think you are getting some OpenCL 1.x headers. Get some recent ones somewhere and set those variables mentioned. This might work:
https://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/16840/intel_sdk_for_opencl_applications_2020.2.487.tar.gz
It's kind of ugly but you can probably just untar it and unpack the appropriate RPMs.

[bader]

Look at this:
-- OpenCL_INCLUDE_DIRS is missing. Will try to download OpenCL headers from github.com
-- OpenCL_LIBRARIES is missing. Will try to download OpenCL ICD Loader from github.com
I think you are getting some OpenCL 1.x headers. Get some recent ones somewhere and set those variables mentioned. This might work:
https://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/16840/intel_sdk_for_opencl_applications_2020.2.487.tar.gz
It's kind of ugly but you can probably just untar it and unpack the appropriate RPMs.

CMake downloads recent version of OpenCL headers and OpenCL ICD loader from Khronos GitHub. These are good for building DPC++ toolchain, but when compiler is invoked, system paths are searched first and if system doesn't have OpenCL headers installed into system other than the version coming with CUDA SDK, there will be conflict. The solution is described in Get Started Guide document - https://intel.github.io/llvm-docs/GetStartedGuide.html#using-the-dpc-toolchain-on-cuda-platforms.

The OpenCL implementation provided by the CUDA SDK is OpenCL 1.2, which is too old to link with the DPC++ runtime and lacks some symbols.

We recommend installing the low level CPU runtime, following the instructions in the next section.

Instead of installing the low level CPU runtime, it is possible to build and install the Khronos ICD loader, which contains all the symbols required.

[bader]

What I don't understand is why there is any reference to cl_ext_intel.h when I build for CUDA. Why does the compiler need to know anything about Intel OpenCL to do a non-OpenCL CUDA build?

I think it might be related to this issue: #1265.

[jeffhammond]

Yeah, it would be good to have no dependency on OpenCL headers -- especially Intel OpenCL extensions -- in the CUDA back-end.

[lfmeadow]

I built clang++ like you did and I’m not seeing the problem. Somehow you are getting a bad CL/cl.h . You should be getting the one in the runtime directory in llvm build, which I am: # 22 "/localdisk/lfmeadow/llvm/build/install/bin/../include/sycl/CL/cl.h" 2 3 Can you run this on the broken build: llvm/build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice -E saxpy.cc > saxpy.i and also: llvm/build/install/bin/clang++ -v -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice saxpy.cc >& log And send me the two output files. The log will give the include paths and the .i file will show the include files as resolved. From: Jeff Hammond <notifications@github.com> Date: Friday, October 16, 2020 at 9:21 AM To: intel/llvm <llvm@noreply.github.com> Cc: Meadows, Lawrence F <lawrence.f.meadows@intel.com>, Comment <comment@noreply.github.com> Subject: Re: [intel/llvm] CUDA back-end fails with OpenCL header problems (#2617) Yeah, it would be good to have no dependency on OpenCL headers -- especially Intel OpenCL extensions -- in the CUDA back-end. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2617 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABH5YK73BVCGWUE36THLPF3SLBXINANCNFSM4SJFUY5A>.

[jeffhammond]

It's picking up /swtools/cuda/latest/include/CL/cl.h all over the place, which is not surprising given I am building for CUDA.

[lfmeadow]

So am I though. It is the order of the -I that is the problem. From: Jeff Hammond <notifications@github.com> Date: Friday, October 16, 2020 at 10:41 AM To: intel/llvm <llvm@noreply.github.com> Cc: Meadows, Lawrence F <lawrence.f.meadows@intel.com>, Comment <comment@noreply.github.com> Subject: Re: [intel/llvm] CUDA back-end fails with OpenCL header problems (#2617) It's picking up /swtools/cuda/latest/include/CL/cl.h all over the place, which is not surprising given I am building for CUDA. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2617 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABH5YK4AMEDXVA4DF33ESFDSLCAT5ANCNFSM4SJFUY5A>.

[jeffhammond]

Okay. But you agree that users should not have deal with this, right? DPC++ should download the OpenCL headers it needs and use those first.

[lfmeadow]

DPC++ already contains the right files, you are picking up the wrong ones. <tl;dr> Something is really screwed up about your build. Part of it is that you are getting the wrong gcc, you are getting the ancient 4.8.5:

…

-internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5 The only way I could fix that was by adding an argument to clang:

--gcc-toolchain=/opt/intel/gcc7.3 The proximate cause though is a bunch of -I that are somehow getting added (You have some environment variable I expect, something containing the string INCLUDE):

-internal-isystem /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl

-I/swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include

-I/swtools/intel/compilers_and_libraries_2017.4.196/linux/ipp/include

-I/swtools/intel/compilers_and_libraries_2017.4.196/linux/mkl/include

-I/swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include

-I/swtools/intel/compilers_and_libraries_2017.4.196/linux/daal/include

-I/swtools/cuda/latest/include

-I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/ipp/include

-I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/mkl/include

-I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/pstl/include

-I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include

-I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include

-I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/daal/include

-I/swtools/ninja/latest/include

-I. You’ve got some pollution from (2 versions of) icc, but the real problem is that this:

-I/swtools/cuda/latest/include contains CL/cl.h Mind you so does this:

-internal-isystem /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl and it comes first so you’d think that clang would find it. -I paths are searched before -internal-isystem paths. From clang -cc1 --help: -internal-isystem <directory> Add directory to the internal system include search path; these are assumed to not be user-provided and are used to model system and standard headers' paths. -I <dir> Add directory to include search path. If there are multiple -I options, these directories are searched in the order they are given before the standard system directories are searched. If the same directory is in the SYSTEM include search paths, for example if also specified with -isystem, the -I option will be ignored So in summary look for some environment variable probably contain the string INCLUDE and unset it, and provide the right path to --gcc-toolchain, and you should compile. It runs as long as you set the plugin type: (py3) [lfmeadow@ortce-skl5 lfmeadow]$ SYCL_DEVICE_FILTER=PI_CUDA ./saxpy.x 4096 SAXPY with 4096 elements Program completed without error. From: Jeff Hammond <notifications@github.com> Date: Friday, October 16, 2020 at 10:45 AM To: intel/llvm <llvm@noreply.github.com> Cc: Meadows, Lawrence F <lawrence.f.meadows@intel.com>, Comment <comment@noreply.github.com> Subject: Re: [intel/llvm] CUDA back-end fails with OpenCL header problems (#2617) Okay. But you agree that users should not have deal with this, right? DPC++ should download the OpenCL headers it needs and use those first. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2617 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABH5YK2ZVPNBOBZ536Q7UCLSLCBC7ANCNFSM4SJFUY5A>.

[jeffhammond]

I've rebuilt DPC++ another 5 times and still cannot get it to pick up the right headers. Whether or not this is solvable, I reject any suggestion that this is something reasonable users are supposed to deal with.

I use GCC 10.2 but it still gets the GCC 4.8 headers, presumably because Linux 3.10 and RHEL/CentOS forces this nonsense on this system.

[jrhammon@pcl-tiergarten dpcpp-tutorial]$ /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang++ -v -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice -L/nfs_home/jrhammon/ISYCL/llvm/build/install/lib -Wl,-rpath=/nfs_home/jrhammon/ISYCL/llvm/build/install/lib saxpy.cc 
clang version 12.0.0 (/nfs_home/jrhammon/ISYCL/llvm/clang 8c667b6745180639c313ea67ad3f175a8212e37b)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /nfs_home/jrhammon/ISYCL/llvm/build/install/bin
Found candidate GCC installation: /usr/lib/gcc/i686-redhat-linux/4.8.2
Found candidate GCC installation: /usr/lib/gcc/i686-redhat-linux/4.8.5
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.2
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.5
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.5
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /data/swtools/cuda/cuda-10.2, version 10.2
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
 "/data/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang-12" -cc1 -triple nvptx64-nvidia-cuda-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2020 -emit-llvm-bc -emit-llvm-uselists -disable-free -main-file-name saxpy.cc -mrelocation-model static -mframe-pointer=all -fno-rounding-math -fno-verbose-asm -no-integrated-as -aux-target-cpu x86-64 -internal-isystem /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl -mlink-builtin-bitcode /data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0/../../clc/libspirv-nvptx64--nvidiacl.bc -mlink-builtin-bitcode /data/swtools/cuda/cuda-10.2/nvvm/libdevice/libdevice.10.bc -target-feature +ptx65 -target-sdk-version=10.2 -target-cpu sm_50 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0 -internal-isystem /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl -I/swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include -I/swtools/intel/compilers_and_libraries_2017.4.196/linux/ipp/include -I/swtools/intel/compilers_and_libraries_2017.4.196/linux/mkl/include -I/swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include -I/swtools/intel/compilers_and_libraries_2017.4.196/linux/daal/include -I/swtools/cuda/latest/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/ipp/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/mkl/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/pstl/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/daal/include -I/swtools/ninja/latest/include -I. -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5 -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/x86_64-redhat-linux -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/backward -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5 -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/x86_64-redhat-linux -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/backward -internal-isystem /usr/local/include -internal-isystem /data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O3 -std=c++17 -fdeprecated-macro -fno-dwarf-directory-asm -fdebug-compilation-dir /nfs_home/jrhammon/dpcpp-tutorial -ferror-limit 3 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -fsycl-unnamed-lambda -o /tmp/saxpy-6fb862.bc -x c++ saxpy.cc
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/swtools/ninja/latest/include"
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include"
ignoring duplicate directory "/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include"
ignoring duplicate directory "/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl"
ignoring duplicate directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5"
ignoring duplicate directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/x86_64-redhat-linux"
ignoring duplicate directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include
 /swtools/intel/compilers_and_libraries_2017.4.196/linux/ipp/include
 /swtools/intel/compilers_and_libraries_2017.4.196/linux/mkl/include
 /swtools/intel/compilers_and_libraries_2017.4.196/linux/daal/include
 /swtools/cuda/latest/include
 /data/swtools/intel/compilers_and_libraries_2019.3.199/linux/ipp/include
 /data/swtools/intel/compilers_and_libraries_2019.3.199/linux/mkl/include
 /data/swtools/intel/compilers_and_libraries_2019.3.199/linux/pstl/include
 /data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include
 /data/swtools/intel/compilers_and_libraries_2019.3.199/linux/daal/include
 .
 /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl
 /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5
 /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/x86_64-redhat-linux
 /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/backward
 /usr/local/include
 /data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0/include
 /usr/include
End of search list.
In file included from saxpy.cc:1:
In file included from ./tutorial.hpp:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:11:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/cl_ext_intel.h:431:9: error: unknown type name 'cl_properties'
typedef cl_properties cl_mem_properties_intel;
        ^
In file included from saxpy.cc:1:
In file included from ./tutorial.hpp:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:121:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/exception.hpp:15:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:228:7: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES'
      CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES,
      ^
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:229:45: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_HOST_PROPERTIES'
  PI_DEVICE_INFO_QUEUE_ON_HOST_PROPERTIES = CL_DEVICE_QUEUE_ON_HOST_PROPERTIES,
                                            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
4 errors generated.