[SYCL] Build fails with "llvm-foreach: No such file or directory"

[krasznaa]

Dear All,

I'm trying to build the latest version of the sycl branch (36c61d7 this morning), but am running into a weird problem while building test executables using the newly created compiler. Like:

[5539/5560] cd /data/projects/intel/llvm/sycl/examples && .../projects/intel/build/tools/sycl/examples/simple-dpcpp-app
FAILED: tools/sycl/examples/CMakeFiles/simple-dpcpp-app_exec tools/sycl/examples/simple-dpcpp-app 
cd /data/projects/intel/llvm/sycl/examples && /data/projects/intel/build/./bin/clang++ -fsycl simple-dpcpp-app.cpp -o /data/projects/intel/build/tools/sycl/examples/simple-dpcpp-app
llvm-foreach: No such file or directory
clang-12: error: llvm-spirv command failed with exit code 1 (use -v to see invocation)

The crazy thing is, this is not reproducible! When I run ninja once more, the build succeeds. Also, if I run that one command in question by itself, it succeeds every time.

So there seems to be some race condition happening. Whenever I see a failure like this, at least two executable creations fail simultaneously. In this example above the other command failing was this one:

FAILED: tools/sycl/unittests/kernel-and-program/CMakeFiles/KernelAndProgramTests_exec tools/sycl/unittests/kernel-and-program/KernelAndProgramTests 
cd /data/projects/intel/llvm/sycl/unittests/kernel-and-program && /data/projects/intel/build/./bin/clang++ -fsycl Cache.cpp KernelRelease.cpp /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./backend/opencl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./backend/level_zero.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/accessor_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/buffer_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/builtins_common.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/builtins_geometric.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/builtins_integer.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/builtins_math.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/builtins_relational.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/pi.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/common.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/config.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/context_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/device_binary_image.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/device_filter.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/device_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/error_handling/enqueue_kernel.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/event_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/filter_selector_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/force_device.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/helpers.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/handler_proxy.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/image_accessor_util.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/image_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/kernel_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/kernel_program_cache.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/memory_manager.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/platform_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/program_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/program_manager/program_manager.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/queue_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/os_util.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/platform_util.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/reduction.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/sampler_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/stream_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/scheduler/commands.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/scheduler/leaves_collection.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/scheduler/scheduler.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/scheduler/graph_processor.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/scheduler/graph_builder.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/spec_constant_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/sycl_mem_obj_t.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/usm/usm_impl.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/util.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./accessor.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./context.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./device.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./device_selector.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./event.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./exception.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./exception_list.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./function_pointer.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./half_type.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./handler.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./interop_handle.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./interop_handler.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./kernel.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./platform.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./program.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./queue.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./sampler.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./stream.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./spirv_ops.cpp.o /data/projects/intel/build/tools/sycl/source/CMakeFiles/sycl_object.dir/./detail/posix_pi.cpp.o -o /data/projects/intel/build/tools/sycl/unittests/kernel-and-program/KernelAndProgramTests -lgtest_main -lgtest -lLLVMSupport -lLLVMTestingSupport -lOpenCL -ldl -nolibsycl -DGTEST_LANG_CXX11=1 -DGTEST_HAS_TR1_TUPLE=0 -D__SYCL_BUILD_SYCL_DLL -I/data/projects/intel/llvm/llvm/utils/unittest/googletest/include -I/data/projects/intel/llvm/llvm/utils/unittest/googlemock/include -I/data/projects/intel/build/include -I/data/projects/intel/llvm/llvm/include -I/data/projects/intel/llvm/sycl/source -I/data/projects/intel/llvm/sycl/unittests -lpthread /usr/lib/x86_64-linux-gnu/libtinfo.so
llvm-foreach: No such file or directory
clang-12: error: llvm-spirv command failed with exit code 1 (use -v to see invocation)

By now I encountered this in multiple setups. Both when building the code natively on Ubuntu 20.04 with the OS's GCC 9, and when building the code inside of CentOS 7/8 Docker images using GCC 8. With the CUDA plugin both enabled and not enabled.

The only explanation I can come up with is that the SYCL code generations from multiple processes are interfering with each other. Maybe by creating files with the same name under /tmp or something similar. 😕

I started seeing this a couple of weeks ago. But by now it's really becoming annoying. 😦

Cheers,
Attila

[bader]

@krasznaa, thanks for reporting this issue. This is definitely a data race in our CMake files.
I guess here the dependencies should list sycl-toolchain rather than just clang and sycl.

[krasznaa]

Interesting. I'll give that a try, as that should be relatively easy to reproduce...

[pgrete]

I'm now seeing the same issue using the intel/oneapi-basekit docker container, see kokkos/kokkos#4679
Adding llvm-foreach to the PATH as suggested by /opt/intel/oneapi/compiler/2022.0.1/linux/bin-llvm/README.txt didn't fix the issue:

[ 44%] Linking CXX executable KokkosExample_query_device
llvm-foreach: No such file or directory
dpcpp: error: gen compiler command failed with exit code 1 (use -v to see invocation)
example/query_device/CMakeFiles/KokkosExample_query_device.dir/build.make:99: recipe for target 'example/query_device/KokkosExample_query_device' failed
make[2]: *** [example/query_device/KokkosExample_query_device] Error 1
CMakeFiles/Makefile2:1371: recipe for target 'example/query_device/CMakeFiles/KokkosExample_query_device.dir/all' failed
make[1]: *** [example/query_device/CMakeFiles/KokkosExample_query_device.dir/all] Error 2
Makefile:145: recipe for target 'all' failed
make: *** [all] Error 2
root@03e9df38e48c:/src/kokkos# which llvm-foreach
/opt/intel/oneapi/compiler/2022.0.1/linux/bin-llvm/llvm-foreach

Am I missing sth?
Also if this (repo/issue) is not the right place to discuss this, it'd be great if you point me in the right direction.
Thanks!
Philipp

[krasznaa]

Hi Philipp,

I imagine yours is a pretty different error from what caused this type of error in my build of the compiler itself. So opening a separate bug would probably be the right way to go about it.

In that ticket you'll probably want to include some more information though. The very least the output from make VERBOSE=1, so that we could see all the build options that make the compiler fail like this.

I imagine that simplifying the problematic code down to something bite-sized will be a bit more of a challenge... 🤔

Cheers,
Attila