[MacOS] directory copy of OpenCL fails

[jeffhammond]

I understand that MacOS is not a priority and that supporting Apple OpenCL is impossible due to lack of SPIR-V. However, it seems that the current testing for OpenCL support is inadequate, as it doesn't determine if the OpenCL CMake finds is sufficient.

I'd like to use POCL for OpenCL CPU support on MacOS, which shouldn't be any different than POCL on Linux. What are the CMake variables to tell the build system to look for POCL instead? I've made many attempts at this but nothing documented on the internet is working.

jrhammon-mac02:build jrhammon$ make VERBOSE=1 sycl-toolchain
/usr/local/Cellar/cmake/3.14.4/bin/cmake -S/Users/jrhammon/Work/OpenCL/ISYCL/llvm/llvm -B/Users/jrhammon/Work/OpenCL/ISYCL/build --check-build-system CMakeFiles/Makefile.cmake 0
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 sycl-toolchain
/usr/local/Cellar/cmake/3.14.4/bin/cmake -S/Users/jrhammon/Work/OpenCL/ISYCL/llvm/llvm -B/Users/jrhammon/Work/OpenCL/ISYCL/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/Cellar/cmake/3.14.4/bin/cmake -E cmake_progress_start /Users/jrhammon/Work/OpenCL/ISYCL/build/CMakeFiles 56
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 tools/sycl/CMakeFiles/sycl-toolchain.dir/all
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f tools/sycl/CMakeFiles/ocl-icd.dir/build.make tools/sycl/CMakeFiles/ocl-icd.dir/depend
cd /Users/jrhammon/Work/OpenCL/ISYCL/build && /usr/local/Cellar/cmake/3.14.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/jrhammon/Work/OpenCL/ISYCL/llvm/llvm /Users/jrhammon/Work/OpenCL/ISYCL/llvm/sycl /Users/jrhammon/Work/OpenCL/ISYCL/build /Users/jrhammon/Work/OpenCL/ISYCL/build/tools/sycl /Users/jrhammon/Work/OpenCL/ISYCL/build/tools/sycl/CMakeFiles/ocl-icd.dir/DependInfo.cmake --color=
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f tools/sycl/CMakeFiles/ocl-icd.dir/build.make tools/sycl/CMakeFiles/ocl-icd.dir/build
[  0%] Copying OpenCL ICD Loader ...
[  0%] Built target ocl-icd
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f tools/sycl/CMakeFiles/ocl-headers.dir/build.make tools/sycl/CMakeFiles/ocl-headers.dir/depend
cd /Users/jrhammon/Work/OpenCL/ISYCL/build && /usr/local/Cellar/cmake/3.14.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/jrhammon/Work/OpenCL/ISYCL/llvm/llvm /Users/jrhammon/Work/OpenCL/ISYCL/llvm/sycl /Users/jrhammon/Work/OpenCL/ISYCL/build /Users/jrhammon/Work/OpenCL/ISYCL/build/tools/sycl /Users/jrhammon/Work/OpenCL/ISYCL/build/tools/sycl/CMakeFiles/ocl-headers.dir/DependInfo.cmake --color=
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f tools/sycl/CMakeFiles/ocl-headers.dir/build.make tools/sycl/CMakeFiles/ocl-headers.dir/build
[  0%] Copying OpenCL headers ...
cd /Users/jrhammon/Work/OpenCL/ISYCL/build/tools/sycl && /usr/local/Cellar/cmake/3.14.4/bin/cmake -E copy_directory /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/OpenCL.framework/CL /Users/jrhammon/Work/OpenCL/ISYCL/build/./lib/clang/9.0.0/include/CL
Error copying directory from "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/OpenCL.framework/CL" to "/Users/jrhammon/Work/OpenCL/ISYCL/build/./lib/clang/9.0.0/include/CL".
make[3]: *** [tools/sycl/CMakeFiles/ocl-headers] Error 1
make[2]: *** [tools/sycl/CMakeFiles/ocl-headers.dir/all] Error 2
make[1]: *** [tools/sycl/CMakeFiles/sycl-toolchain.dir/rule] Error 2
make: *** [sycl-toolchain] Error 2

Steps to reproduce

#!/bin/bash

export SYCL_HOME=$HOME/Work/OpenCL/ISYCL/llvm

mkdir -p $SYCL_HOME

cd $SYCL_HOME && \
    time git pull || \
    time git clone https://github.com/intel/llvm.git $SYCL_HOME

rm -rf $SYCL_HOME/../build

mkdir -p $SYCL_HOME/../build && \
    cd $SYCL_HOME/../build && \
    time cmake \
        -DCMAKE_INSTALL_PREFIX=/opt/isycl \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS="clang;llvm-spirv;sycl" \
        -DLLVM_EXTERNAL_PROJECTS="llvm-spirv;sycl" \
        -DLLVM_EXTERNAL_SYCL_SOURCE_DIR=$SYCL_HOME/sycl \
        -DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=$SYCL_HOME/llvm-spirv \
        -DLLVM_TOOL_SYCL_BUILD=ON \
        -DLLVM_TOOL_LLVM_SPIRV_BUILD=ON \
        $SYCL_HOME/llvm

time make -j4 sycl-toolchain

[AlexeySachkov]

Hi @jeffhammond,

As I can see here, OpenCL is queried using standard cmake functionality find_package.

It is expected that this function will define several variables like:

• OpenCL_INCLUDE_DIRS
• OpenCL_LIBRARIES

If something wasn't found, then script tries to download and build corresponding components.

From your logs it looks like OpenCL_INCLUDE_DIRS was defined by find_package, but its content might be wrong:

Error copying directory from "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/OpenCL.framework/CL" to "/Users/jrhammon/Work/OpenCL/ISYCL/build/./lib/clang/9.0.0/include/CL".

I think you can try to pass -DOpenCL_INCLUDE_DIRS=/path/to/opencl/includes manually to cmake to set correct path to OpenCL header files.

Note, that copy command expects CL directory to be present in directory pointed by OpenCL_INLUDE_DIRS:

llvm/sycl/CMakeLists.txt

Line 58 in f0491f0

COMMAND ${CMAKE_COMMAND} -E copy_directory ${OpenCL_INCLUDE_DIRS}/CL ${dst_dir}/CL

[jeffhammond]

Unfortunately, the "find OpenCL" feature in CMake does not respect those options and insists on finding Apple OpenCL rather than failing when told to explicitly look in an invalid director, as configure would.

CMake invocation from clean directory

jrhammon-mac02:build jrhammon$ rm -rf *
jrhammon-mac02:build jrhammon$ time cmake -DCMAKE_INSTALL_PREFIX=/opt/isycl -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;llvm-spirv;sycl" -DLLVM_EXTERNAL_PROJECTS="llvm-spirv;sycl" -DLLVM_EXTERNAL_SYCL_SOURCE_DIR=$SYCL_HOME/sycl -DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=$SYCL_HOME/llvm-spirv -DLLVM_TOOL_SYCL_BUILD=ON -DLLVM_TOOL_LLVM_SPIRV_BUILD=ON $SYCL_HOME/llvm -DOpenCL_INCLUDE_DIRS=/tmp -DOpenCL_LIBRARIES=/tmp $SYCL_HOME/llvm && make sycl-toolchain

Excerpt from CMake output

-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2") 
-- Checking for one of the modules 'SPIRV-Tools'
CMake Warning at /Users/jrhammon/Work/OpenCL/ISYCL/llvm/llvm-spirv/test/CMakeLists.txt:21 (message):
  spirv-val not found! SPIR-V generated for test suite will not be validated.


-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - found
-- Found OpenCL: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/OpenCL.framework (found version "1.2") 
OpenCL header have been found under /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/OpenCL.framework.
OpenCL loader has been found: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/OpenCL.framework.

[AlexeySachkov]

It actually works on my Linux machine, but looking at sources of FindOpenCL module it probably shouldn't.

What else you can try:

• hacky workaround, but definitely should work - remove find_package call in sycl/CMakeLists.txt
• As I can see here and here, cmake will try to use some env variable as hints where OpenCL is located (note: list of env variables is different for headers and library (I don't know why))
• Take a look at documentation for find_path and find_library functions: they are used by FindOpenCL.cmake and their behavior can be controlled via bunch of cmake options.

I also recommend to find FindOpenCL.cmake locally to check what it uses and which env variables are listed in it as hints for searching.

I think that's all that I can suggest right now

[jeffhammond]

That sort of works, but the directory copy incorrectly assumes that Apple OpenCL headers are in ./CL ...

Error copying directory from "/Library/Developer/CommandLineTools//SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers//CL" to "/Users/jrhammon/Work/OpenCL/ISYCL/build/./lib/clang/9.0.0/include/CL".

...even though CMake rightly detects that they are not...

OpenCL header have been found under /Library/Developer/CommandLineTools//SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers/.

The fact that Apple alone doesn't put their OpenCL headers in ./CL is well-known and every OpenCL project I've read has preprocessor logic to account for this.

Full Output

jrhammon-mac02:build jrhammon$ time cmake -DCMAKE_INSTALL_PREFIX=/opt/isycl -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;llvm-spirv;sycl" -DLLVM_EXTERNAL_PROJECTS="llvm-spirv;sycl" -DLLVM_EXTERNAL_SYCL_SOURCE_DIR=$SYCL_HOME/sycl -DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=$SYCL_HOME/llvm-spirv -DLLVM_TOOL_SYCL_BUILD=ON -DLLVM_TOOL_LLVM_SPIRV_BUILD=ON $SYCL_HOME/llvm -DOpenCL_INCLUDE_DIRS=/Library/Developer/CommandLineTools//SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers/ -DOpenCL_LIBRARIES=/tmp $SYCL_HOME/llvm && time make sycl-toolchain
-- Found libtool - /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool
-- clang project is enabled
-- clang-tools-extra project is disabled
-- compiler-rt project is disabled
-- debuginfo-tests project is disabled
-- libclc project is disabled
-- libcxx project is disabled
-- libcxxabi project is disabled
-- libunwind project is disabled
-- lld project is disabled
-- lldb project is disabled
-- llgo project is disabled
-- openmp project is disabled
-- parallel-libs project is disabled
-- polly project is disabled
-- pstl project is disabled
-- llvm-spirv project is enabled
-- sycl project is enabled
-- Could NOT find Z3: Found unsuitable version "0.0.0", but required is at least "4.7.1" (found Z3_LIBRARIES-NOTFOUND)
-- Native target architecture is X86
-- Threads enabled.
-- Doxygen disabled.
-- Go bindings disabled.
-- Found ld64 - /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH) 
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH) 
-- OCaml bindings disabled.
-- LLVM host triple: x86_64-apple-darwin18.6.0
-- LLVM default target triple: x86_64-apple-darwin18.6.0
-- Building with -fPIC
-- Constructing LLVMBuild project information
-- Linker detection: ld64
-- Targeting AArch64
-- Targeting AMDGPU
-- Targeting ARM
-- Targeting BPF
-- Targeting Hexagon
-- Targeting Lanai
-- Targeting Mips
-- Targeting MSP430
-- Targeting NVPTX
-- Targeting PowerPC
-- Targeting Sparc
-- Targeting SystemZ
-- Targeting WebAssembly
-- Targeting X86
-- Targeting XCore
-- Clang version: 9.0.0
-- Performing Test LINKER_ORDER_FILE_WORKS
-- Performing Test LINKER_ORDER_FILE_WORKS - Success
-- Checking for one of the modules 'SPIRV-Tools'
CMake Warning at /Users/jrhammon/Work/OpenCL/ISYCL/llvm/llvm-spirv/test/CMakeLists.txt:21 (message):
  spirv-val not found! SPIR-V generated for test suite will not be validated.


OpenCL header have been found under /Library/Developer/CommandLineTools//SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers/.
OpenCL loader has been found: /tmp.
-- Failed to find LLVM FileCheck
-- git Version: v0.0.0-dirty
-- Version: 0.0.0
-- Performing Test HAVE_THREAD_SAFETY_ATTRIBUTES -- failed to compile
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Configuring done
WARNING: Target "sycl" requests linking to directory "/tmp".  Targets may link only to libraries.  CMake is dropping the item.
WARNING: Target "sycl-check" requests linking to directory "/tmp".  Targets may link only to libraries.  CMake is dropping the item.
WARNING: Target "sycl-check" requests linking to directory "/tmp".  Targets may link only to libraries.  CMake is dropping the item.
WARNING: Target "get_device_count_by_type" requests linking to directory "/tmp".  Targets may link only to libraries.  CMake is dropping the item.
-- Generating done
-- Build files have been written to: /Users/jrhammon/Work/OpenCL/ISYCL/build

real	0m39.209s
user	0m14.632s
sys	0m17.659s
[  0%] Copying OpenCL ICD Loader ...
[  0%] Built target ocl-icd
[  0%] Copying OpenCL headers ...
Error copying directory from "/Library/Developer/CommandLineTools//SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers//CL" to "/Users/jrhammon/Work/OpenCL/ISYCL/build/./lib/clang/9.0.0/include/CL".
make[3]: *** [tools/sycl/CMakeFiles/ocl-headers] Error 1
make[2]: *** [tools/sycl/CMakeFiles/ocl-headers.dir/all] Error 2
make[1]: *** [tools/sycl/CMakeFiles/sycl-toolchain.dir/rule] Error 2
make: *** [sycl-toolchain] Error 2

real	0m5.561s
user	0m4.615s
sys	0m0.275s
jrhammon-mac02:build jrhammon$ ll /Library/Developer/CommandLineTools//SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers//CL
ls: /Library/Developer/CommandLineTools//SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers//CL: No such file or directory
jrhammon-mac02:build jrhammon$ ll /Library/Developer/CommandLineTools//SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers/CL
ls: /Library/Developer/CommandLineTools//SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers/CL: No such file or directory
jrhammon-mac02:build jrhammon$ find  /Library/Developer/CommandLineTools/ -name cl.h
/Library/Developer/CommandLineTools//SDKs/MacOSX.sdk 1/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers/cl.h
/Library/Developer/CommandLineTools//SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework/Versions/A/Headers/cl.h

[alexbatashev]

@jeffhammond just comment out find_package in CMakeLists.txt. Both ICD and Headers will be downloaded from Khronos repo. A bit hacky, but seems to be working for me. Please note, that ICD is not your biggest problem.

1. Apple's default compiler is Apple Clang 10. Which is a little bit different from vanilla clang. As well as gcc 5.5.0 which is a requirement for now. The biggest problem you might face is libc++ and GNU ifunc. Currently we do not support using libc++ (see ICE when using libcxx #186). The second problem seems to be resolved by [SYCL] Start running OpenCL through PI #281. Anyway, this kind of issues is going to be addressed while porting SYCL compiler to Windows.
2. Second problem is much bigger. You need either OpenCL 2.1+ or the installation that supports cl_khr_il_program extension. AFAIK, neither Apple, nor PoCL support it. Our implementation heavily relies on SPIRV. And while Apple has abandoned their OpenCL driver, one could make use of SPIRV translator for PoCL. But Intel has committed a lot of bufixes and improvements to translator. And as they were merged to master branch, you'll most likely will have to build PoCL and the translator from top of the trunk. I'm not sure this is an easy task.

[agozillon]

Possibly not too interesting or helpful, but it may be interesting to someone. However, you could in theory target POCL with 2 step compile and some tinkering with the Xilinx implementation variation as it's SPIR-df (and by extension other SPIR devices I imagine). I did this ages ago before adding our own driver in, but it won't work out of the box anymore without proper usage of some LLVM passes on the kernel and the right defines passed to the device compilation, but it wouldn't be too hard I think and I'd be happy to help out if someone wants to give it a try...

The main caveat with that is, you're stuck with our implementation! And we live downstream with a varied amount of delay so it doesn't progress anywhere near as fast. And we also haven't replaced a lot of the SPIRV builtins with SPIR builtins yet (in some cases it's not as easy as dropping some SPIR builtins in place, notably I think sub-groups).

I think @keryell intends to eventually add support directly into the compiler via the driver for POCL/non-Xilinx SPIR devices, but that won't happen for a while I think. It's a little low down on the priority list for us at the moment..

I think alternatively with two step compilation you can quite easily get some SPIR-df kernels for POCL consumption off the ground if you're willing to stick some SPIR builtins (as long as they're global scope, the mangling should work out fine..) in place of the SIMT SPIRV builtins and hack up the runtime in a few places.

[jeffhammond]

@alexbatashev I really only want the compiler to work on the CPU, so I don't see why I need Apple OpenCL. Frankly, I shouldn't need a runtime or SPIR-V for the host device. At some point, we need to seriously think about portability on the CPU side to all operating systems, which means implementing the host and cpu devices with a portable runtime with minimal dependencies. I know TBB serves this purpose inside OpenCL, but it should be used directly to avoid being blocked by the inadequate OpenCL ecosystem.

[jeffhammond]

And to be clear, I wouldn't be in this position if triSYCL was feature-complete, but it's not and that doesn't appear to be a goal (no progress on triSYCL/triSYCL#66 in nearly two years).

[alexbatashev]

@jeffhammond ok, I see your point. AFAIK, host device is heavy WIP right now. But it will be implemented eventually as it is required by spec. I see 3 major blockers for SYCL on macOS:

1. You still need up to date ICD and headers. Even if you don't actually use OpenCL, the runtime does. And you need to be able to link this whole thing somehow. I'm not sure if you can build one today, but that should be solvable. Actually, I'm curious. I'll give it a try on weekend and let you know.
2. libc++ support. Tough thing. Work in progress.
3. Host device itself. There is still a lot to do. If I'm not mistaken, it executes kernels in 1 thread. And this is also an interesting topic to discuss. There are many ways to implement multithreading. From POSIX to TBB. And it will be hard to please everyone. Depending on that decision, we may or may not get macOS compatible host device.

Am I right that you don't cate about performance on macOS? If you only intend to use your macOS environment for development/debugging, and run actual workload on linux/windows, that should be relatively easy. SYCL is currently unstable, but it won't last forever. If you want to do some heavy computation on your Mac, I'm going to disappoint you. You won't get much performance. In fact, you may get significant performance regressions.

[keryell]

@alexbatashev the CPU support should be quite simple compared to anything else, if you make the assumption that you do not support barriers and write everything in the runtime in plain modern C++ without any specific compiler support.

I have updated the documentation of triSYCL in triSYCL/triSYCL#252 explaining it.

Actually it was implemented by @jeffamstutz from Intel, so you know who can do the work! ;-)

[alexbatashev]

Finally, I could build SYCL compiler on macOS. The majority of problems lies inside RT. There's quite plenty of work to make it portable. Anyway, in spite of tons of warnings I was able to build it. I failed to build tests, but I haven't analysed the problem yet.
Step by step guide:

1. Apply my set of hacky hacks
2. cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_EXTERNAL_PROJECTS="llvm-spirv;sycl" -DLLVM_EXTERNAL_SYCL_SOURCE_DIR=$SYCL_HOME/llvm/sycl -DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=$SYCL_HOME/llvm/llvm-spirv -DLLVM_ENABLE_PROJECTS="clang;llvm-spirv;sycl" $SYCL_HOME/llvm
3. cmake --build . --target check-sycl -- -j4

Set of hacks:
mac.txt

[jeffhammond]

Thanks! This is great. I will see if I can figure out some of the errors.

[alexbatashev]

BTW, this is sample of errors I see:

...
llvm/build/libcxx/lib/clang/9.0.0/include/CL/__spirv/spirv_types.hpp:11:10: fatal error: 'cstdint' file not found
#include <cstdint>
         ^~~~~~~~~
1 error generated.
...

They are the same for every single test. Most likely will be resolved by env variables or something like this.

[jeffhammond]

Ah that’s pretty common with Clang on Apple it it’s not configured to know where Apple headers are.