OpenCl compiling issue

Hi, I'm trying to compile llama.cpp using my opencl drivers. My device is a Samsung s10+ with termux.

On downloading and attempting make with LAMA_CLBLAST=1, I receive an error:

ggml-opencl.cpp:8:10: fatal error: 'clblast.h' file not found
#include <clblast.h>

I edited the ggml-open.cl.cpp file TRYING to point it to my opencl libraries by replacing <clblast.h> with ocl_icd.h. (as my library path is /data/data/com.termux/files/usr/include)

Then with make LLAMA_CLBLAST=1 I received this:

I llama.cpp build info:
I UNAME_S:  Linux
I UNAME_P:  unknown
I UNAME_M:  aarch64
I CFLAGS:   -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -DGGML_USE_CLBLAST
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_CLBLAST
I LDFLAGS:  -lclblast -lOpenCL
I CC:       clang version 16.0.4
I CXX:      clang version 16.0.4                      
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
cc  -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -DGGML_USE_CLBLAST   -c ggml.c -o ggml.o
ggml.c:2154:5: warning: implicit conversion increases floating-point precision: 'float32_t' (aka 'float') to 'ggml_float' (aka 'double') [-Wdouble-promotion]         GGML_F16_VEC_REDUCE(sumf, sum);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ggml.c:1696:41: note: expanded from macro 'GGML_F16_VEC_REDUCE'                                                 #define GGML_F16_VEC_REDUCE         GGML_F32Cx4_REDUCE
                                        ^
ggml.c:1686:38: note: expanded from macro 'GGML_F32Cx4_REDUCE'
    #define GGML_F32Cx4_REDUCE       GGML_F32x4_REDUCE
                                     ^
ggml.c:1619:11: note: expanded from macro 'GGML_F32x4_REDUCE'
    res = GGML_F32x4_REDUCE_ONE(x[0]);         \
        ~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~
ggml.c:1607:34: note: expanded from macro 'GGML_F32x4_REDUCE_ONE'
#define GGML_F32x4_REDUCE_ONE(x) vaddvq_f32(x)
                                 ^~~~~~~~~~~~~
ggml.c:3196:9: warning: implicit conversion increases floating-point precision: 'float32_t' (aka 'float') to 'ggml_float' (aka 'double') [-Wdouble-promotion]
        GGML_F16_VEC_REDUCE(sumf[k], sum[k]);
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~          ggml.c:1696:41: note: expanded from macro 'GGML_F16_VEC_REDUCE'
    #define GGML_F16_VEC_REDUCE         GGML_F32Cx4_REDUCE                                                                                          ^
ggml.c:1686:38: note: expanded from macro 'GGML_F32Cx4_REDUCE'
    #define GGML_F32Cx4_REDUCE       GGML_F32x4_REDUCE                                     ^
ggml.c:1619:11: note: expanded from macro 'GGML_F32x4_REDUCE'
    res = GGML_F32x4_REDUCE_ONE(x[0]);         \              ~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~
ggml.c:1607:34: note: expanded from macro 'GGML_F32x4_REDUCE_ONE'
#define GGML_F32x4_REDUCE_ONE(x) vaddvq_f32(x)                                         ^~~~~~~~~~~~~
2 warnings generated.
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_CLBLAST -c llama.cpp -o llama.o
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_CLBLAST -c examples/common.cpp -o common.o
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_CLBLAST -c ggml-opencl.cpp -o ggml-opencl.o
ggml-opencl.cpp:694:13: error: use of undeclared identifier 'clblast'                                                   clblast::StatusCode status = clblast::Gemm<cl_float>(clblast::Layout::kColMajor,
            ^
ggml-opencl.cpp:694:42: error: use of undeclared identifier 'clblast'                                                   clblast::StatusCode status = clblast::Gemm<cl_float>(clblast::Layout::kColMajor,                                                         ^
ggml-opencl.cpp:694:56: error: unexpected type name 'cl_float': expected expression                                     clblast::StatusCode status = clblast::Gemm<cl_float>(clblast::Layout::kColMajor,
                                                       ^
ggml-opencl.cpp:694:66: error: use of undeclared identifier 'clblast'                                                   clblast::StatusCode status = clblast::Gemm<cl_float>(clblast::Layout::kColMajor,
                                                                 ^
ggml-opencl.cpp:695:56: error: use of undeclared identifier 'clblast'                                                                                              clblast::Transpose::kYes, clblast::Transpose::kNo,
                                                       ^
ggml-opencl.cpp:695:82: error: use of undeclared identifier 'clblast'                                                                                              clblast::Transpose::kYes, clblast::Transpose::kNo,
                                                                                 ^
ggml-opencl.cpp:704:27: error: use of undeclared identifier 'clblast'                                                   if (status != clblast::StatusCode::kSuccess) {
                          ^                           ggml-opencl.cpp:798:13: error: use of undeclared identifier 'clblast'
            clblast::StatusCode status = clblast::Gemm<cl_half>(clblast::Layout::kColMajor,                             ^
ggml-opencl.cpp:798:42: error: use of undeclared identifier 'clblast'
            clblast::StatusCode status = clblast::Gemm<cl_half>(clblast::Layout::kColMajor,                                                          ^            ggml-opencl.cpp:798:56: error: unexpected type name 'cl_half': expected expression                                      clblast::StatusCode status = clblast::Gemm<cl_half>(clblast::Layout::kColMajor,                                                                        ^                                                    ggml-opencl.cpp:798:65: error: use of undeclared identifier 'clblast'
            clblast::StatusCode status = clblast::Gemm<cl_half>(clblast::Layout::kColMajor,
                                                                ^
ggml-opencl.cpp:799:56: error: use of undeclared identifier 'clblast'                                                                                              clblast::Transpose::kYes, clblast::Transpose::kNo,                                                          ^                                                    ggml-opencl.cpp:799:82: error: use of undeclared identifier 'clblast'                                                                                              clblast::Transpose::kYes, clblast::Transpose::kNo,
                                                                                 ^
ggml-opencl.cpp:808:27: error: use of undeclared identifier 'clblast'
            if (status != clblast::StatusCode::kSuccess) {                                                                            ^
ggml-opencl.cpp:910:17: error: use of undeclared identifier 'clblast'
                clblast::StatusCode status = clblast::Gemm<cl_float>(clblast::Layout::kColMajor,
                ^                                     ggml-opencl.cpp:910:46: error: use of undeclared identifier 'clblast'
                clblast::StatusCode status = clblast::Gemm<cl_float>(clblast::Layout::kColMajor,
                                             ^        ggml-opencl.cpp:910:60: error: unexpected type name 'cl_float': expected expression                                         clblast::StatusCode status = clblast::Gemm<cl_float>(clblast::Layout::kColMajor,
                                                           ^
ggml-opencl.cpp:910:70: error: use of undeclared identifier 'clblast'
                clblast::StatusCode status = clblast::Gemm<cl_float>(clblast::Layout::kColMajor,
                                                                     ^
ggml-opencl.cpp:911:60: error: use of undeclared identifier 'clblast'
                                                           clblast::Transpose::kYes, clblast::Transpose::kNo,                                                                                                                ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]                                         20 errors generated.
make: *** [Makefile:150: ggml-opencl.o] Error 1

Current Behavior

It appears my libraries for opencl are not included and I don't know how to make llama.cpp recognize them during compilation.

clinfo:

Number of platforms                               1
  Platform Name                                   clvk
  Platform Vendor                                 clvk
  Platform Version                                OpenCL 3.0 clvk
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd cl_khr_extended_versioning
  Platform Extensions with Version                cl_khr_icd                                                       0x400000 (1.0.0)
                                                  cl_khr_extended_versioning                                       0x400000 (1.0.0)
  Platform Numeric Version                        0xc00000 (3.0.0)
  Platform Extensions function suffix             clvk
  Platform Host timer resolution                  0ns

  Platform Name                                   clvk
Number of devices                                 1
  Device Name                                     Adreno (TM) 640
  Device Vendor                                   FIXME
  Device Vendor ID                                0x5143
  Device Version                                  OpenCL 3.0 CLVK on Vulkan v1.1.128 driver 2149539840
  Device UUID                                     43510000-0500-0000-8002-500014008002
  Driver UUID                                     02000000-0000-0000-0000-000000000000
  Valid Device LUID                               No
  Device LUID                                     0000-000000000000
  Device Node Mask                                0
  Device Numeric Version                          0xc00000 (3.0.0)
  Driver Version                                  3.0 CLVK on Vulkan v1.1.128 driver 2149539840
  Device OpenCL C Version                         OpenCL C 1.2 CLVK on Vulkan v1.1.128 driver 2149539840
  Device OpenCL C Numeric Version                 0x402000 (1.2.0)
  Device OpenCL C all versions                    OpenCL C                                                         0x400000 (1.0.0)
                                                  OpenCL C                                                         0x401000 (1.1.0)
                                                  OpenCL C                                                         0x402000 (1.2.0)
                                                  OpenCL C                                                         0xc00000 (3.0.0)
  Device OpenCL C features                        __opencl_c_images                                                0xc00000 (3.0.0)
                                                  __opencl_c_read_write_images                                     0xc00000 (3.0.0)
                                                  __opencl_c_3d_image_writes                                       0xc00000 (3.0.0)
                                                  __opencl_c_atomic_order_acq_rel                                  0xc00000 (3.0.0)
                                                  __opencl_c_atomic_scope_device                                   0xc00000 (3.0.0)
                                                  __opencl_c_subgroups                                             0xc00000 (3.0.0)
  Latest conformance test passed                  FIXME
  Device Type                                     GPU, Default
  Device Profile                                  FULL_PROFILE
  Device Available                                Yes
  Compiler Available                              Yes
  Linker Available                                Yes
  Max compute units                               2
  Max clock frequency                             0MHz
  Device Partition                                (core)
    Max number of sub-devices                     0
    Supported partition types                     None
    Supported affinity domains                    (n/a)
  Max work item dimensions                        3
  Max work item sizes                             1024x1024x64
  Max work group size                             1024
  Preferred work group size multiple (device)     16
  Preferred work group size multiple (kernel)     16
  Max sub-groups per work group                   16
  Preferred / native vector sizes
    char                                                 1 / 1
    short                                                1 / 1
    int                                                  1 / 1
    long                                                 1 / 1
    half                                                 1 / 1        (n/a)
    float                                                1 / 1
    double                                               1 / 1        (n/a)
  Half-precision Floating-point support           (n/a)
  Single-precision Floating-point support         (core)
    Denormals                                     No
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 No
    Round to infinity                             No
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Double-precision Floating-point support         (n/a)
  Address bits                                    32, Little-Endian
  Global memory size                              2147483648 (2GiB)
  Error Correction support                        No
  Max memory allocation                           536870912 (512MiB)
  Unified memory for Host and Device              Yes
  Shared Virtual Memory (SVM) capabilities        (core)
    Coarse-grained buffer sharing                 No
    Fine-grained buffer sharing                   No
    Fine-grained system sharing                   No
    Atomics                                       No
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       1024 bits (128 bytes)
  Preferred alignment for atomics
    SVM                                           0 bytes
    Global                                        0 bytes
    Local                                         0 bytes
  Atomic memory capabilities                      relaxed, acquire/release, work-group scope, device scope
  Atomic fence capabilities                       relaxed, acquire/release, work-item scope, work-group scope, device scope
  Max size for global variable                    0
  Preferred total size of global vars             0
  Global Memory cache type                        None
  Image support                                   Yes
    Max number of samplers per kernel             20
    Max size for 1D images from buffer            16384 pixels
    Max 1D or 2D image array size                 2048 images
    Base address alignment for 2D image buffers   0 bytes
    Pitch alignment for 2D image buffers          0 pixels
    Max 2D image size                             16384x16384 pixels
    Max 3D image size                             2048x2048x2048 pixels
    Max number of read image args                 524288
    Max number of write image args                524288
    Max number of read/write image args           524288
  Pipe support                                    No
  Max number of pipe args                         0
  Max active pipe reservations                    0
  Max pipe packet size                            0
  Local memory type                               Local
  Local memory size                               32768 (32KiB)
  Max number of constant args                     8
  Max constant buffer size                        65536 (64KiB)
  Generic address space support                   No
  Max size of kernel argument                     1024
  Queue properties (on host)
    Out-of-order execution                        No
    Profiling                                     Yes
  Device enqueue capabilities                     (n/a)
  Queue properties (on device)
    Out-of-order execution                        No
    Profiling                                     No
    Preferred size                                0
    Max size                                      0
  Max queues on device                            0
  Max events on device                            0
  Prefer user sync for interop                    Yes
  Profiling timer resolution                      1ns
  Execution capabilities
    Run OpenCL kernels                            Yes
    Run native kernels                            No
    Non-uniform work-groups                       Yes
    Work-group collective functions               No
    Sub-group independent forward progress        No
    IL version                                    SPIR-V_1.0
    ILs with version                              SPIR-V                                                           0x400000 (1.0.0)
  printf() buffer size                            1048576 (1024KiB)
  Built-in kernels                                (n/a)
  Built-in kernels with version                   (n/a)
  Device Extensions                               cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_extended_versioning cl_khr_create_command_queue cl_khr_il_program cl_khr_spirv_no_integer_wrap_decoration cl_arm_non_uniform_work_group_size cl_khr_suggested_local_work_size cl_khr_3d_image_writes cl_khr_device_uuid
  Device Extensions with Version                  cl_khr_global_int32_base_atomics                                 0x400000 (1.0.0)
                                                  cl_khr_global_int32_extended_atomics                             0x400000 (1.0.0)
                                                  cl_khr_local_int32_base_atomics                                  0x400000 (1.0.0)
                                                  cl_khr_local_int32_extended_atomics                              0x400000 (1.0.0)
                                                  cl_khr_byte_addressable_store                                    0x400000 (1.0.0)
                                                  cl_khr_extended_versioning                                       0x400000 (1.0.0)
                                                  cl_khr_create_command_queue                                      0x400000 (1.0.0)
                                                  cl_khr_il_program                                                0x400000 (1.0.0)
                                                  cl_khr_spirv_no_integer_wrap_decoration                          0x400000 (1.0.0)
                                                  cl_arm_non_uniform_work_group_size                               0x400000 (1.0.0)
                                                  cl_khr_suggested_local_work_size                                 0x400000 (1.0.0)
                                                  cl_khr_3d_image_writes                                           0x400000 (1.0.0)
                                                  cl_khr_device_uuid                                               0x400000 (1.0.0)

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  clvk
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [clvk]
  clCreateContext(NULL, ...) [default]            Success [clvk]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  Success (1)
    Platform Name                                 clvk
    Device Name                                   Adreno (TM) 640
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (1)
    Platform Name                                 clvk
    Device Name                                   Adreno (TM) 640
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (1)
    Platform Name                                 clvk
    Device Name                                   Adreno (TM) 640

ICD loader properties
  ICD loader Name                                 OpenCL ICD Loader
  ICD loader Vendor                               OCL Icd free software
  ICD loader Version                              2.3.1
  ICD loader Profile                              OpenCL 3.0

lscpu:

Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 8
  On-line CPU(s) list:  0-7
Vendor ID:              Qualcomm
  Model name:           Kryo-4XX-Silver
    Model:              14
    Thread(s) per core: 1
    Core(s) per socket: 4
    Socket(s):          1
    Stepping:           0xd
    CPU(s) scaling MHz: 62%
    CPU max MHz:        1785.6000
    CPU min MHz:        300.0000
    BogoMIPS:           38.40
    Flags:              fp asimd evtstrm aes pmull sha
                        1 sha2 crc32 atomics fphp asim
                        dhp cpuid asimdrdm lrcpc dcpop
                         asimddp
  Model name:           Kryo-4XX-Gold
    Model:              14
    Thread(s) per core: 1
    Core(s) per socket: 2
    Socket(s):          2
    Stepping:           0xd
    CPU(s) scaling MHz: 71%
    CPU max MHz:        2841.6001
    CPU min MHz:        710.4000
    BogoMIPS:           38.40
    Flags:              fp asimd evtstrm aes pmull sha
                        1 sha2 crc32 atomics fphp asim
                        dhp cpuid asimdrdm lrcpc dcpop
                         asimddp

clpeak:

                                                      Platform: clvk                                          Device: Adreno (TM) 640
    Driver version  : 3.0 CLVK on Vulkan v1.1.128 driver 2149539840 (Android)                                   Compute units   : 2                                   Clock frequency : 0 MHz
                                                          Global memory bandwidth (GBPS)                          float   : 21.86
      float2  : 24.10
      float4  : 19.43
      float8  : 10.23
      float16 : 8.94
                                                          Single-precision compute (GFLOPS)
      float   : 369.29
      float2  : 273.19
      float4  : 309.08                                      float8  : 507.69
      float16 : 523.76

    No half precision support! Skipped                
    No double precision support! Skipped

    Integer compute (GIOPS)                                 int   : 109.64
      int2  : 71.84
      int4  : 139.36
      int8  : 80.51                                         int16 : 78.88

    Integer compute Fast 24bit (GIOPS)
      int   : 108.55                                        int2  : 71.70
      int4  : 139.01
      int8  : 80.41
      int16 : 77.72                                   
    Transfer bandwidth (GBPS)
      enqueueWriteBuffer              : 8.22
      enqueueReadBuffer               : 1.04                enqueueWriteBuffer non-blocking : 8.67
      enqueueReadBuffer non-blocking  : 1.05
      enqueueMapBuffer(for read)      : 8992.81
        memcpy from mapped ptr        : 1.04                enqueueUnmap(after write)       : 58355.54
        memcpy to mapped ptr          : 8.60

    Kernel launch latency : 27.10 us

Thanks for any direction on this matter.

[SlyEcho]

Try using CMake instead, it is much better at finding libraries and even then it can also be better manually configured in cases when it doesn't find something at first.

[ekolawole]

how to run in CLI with cmake?

Try using CMake instead, it is much better at finding libraries and even then it can also be better manually configured in cases when it doesn't find something at first.

Thanks for your response. It appears to have compiled, but now I can't run ./main as it says no command found.

Is there anyone that can assist me with compiling so that I can use ./main?

[SlyEcho]

With CMake main is in the subdirectory bin of the build directory.

With CMake main is in the subdirectory bin of the build directory.

Lovely, thank you for the direction. I can run ./main from the bin subfolder.

It appears clblast does not have a system_info label like openBlas does (llama.cpp shows BLAS=1 when compiled with openBlas), so I'll try and test another way to see if my GPU is engaged.

To clarify, clblast is an alternative to openblas, is that right?

I assume I can't run both openBlas and Clblast at the same time, but maybe I'm missing something.

[SlyEcho]

It seems like it was not compiled in, then. It should show which platform and device it uses on start up and BLAS = 1 should also show. You need to turn on the LLAMA_CLBLAST option, you can do that on the command line with -DLLAMA_CLBLAST=ON when running CMake or editing the CMakeCache.txt or using a tool such as ccmake.

Thanks again for the information.

I am trying to compile using cmake . -DLLAMA_CLBLAST=ON

CMake Warning at CMakeLists.txt:210 (message):
  CLBlast not found

Neither make nor cmake find the library, so I'm still uncertain how to actually point llama.cpp to my libraries in /data/data/com.termux/files/usr/include/CL

Edit: to clarify, editing line in cMakeCache.txt,

CLBlast_DIR:PATH=CLBlast_DIR-NOTFOUND

To

CLBlast_DIR:PATH /data/data/com.termux/files/usr/include/CL

And then trying cmake . -DLLAMA_CLBLAST=ON gives me this:

CMake Warning at CMakeLists.txt:107 (message):
  Git repository not found; to enable automatic generation of build info,
  make sure Git is installed and the project is a Git repository.


CMake Warning at CMakeLists.txt:200 (find_package):
  By not providing "FindCLBlast.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "CLBlast", but
  CMake did not find one.

  Could not find a package configuration file provided by "CLBlast" with any
  of the following names:

    CLBlastConfig.cmake
    clblast-config.cmake

  Add the installation prefix of "CLBlast" to CMAKE_PREFIX_PATH or set
  "CLBlast_DIR" to a directory containing one of the above files.  If
  "CLBlast" provides a separate development package or SDK, be sure it has
  been installed.


CMake Warning at CMakeLists.txt:210 (message):
  CLBlast not found


-- CMAKE_SYSTEM_PROCESSOR: aarch64
-- ARM detected
-- Configuring done (0.1s)
-- Generating done (0.1s)
-- Build files have been written to: /data/data/com.termux/files/home/newllama

[SlyEcho]

CLBlast_DIR is supposed to point at CLBlast's CMake files, on my system it is /usr/local/lib/cmake/CLBlast.

You can try to use CMAKE_PREFIX_PATH (environment variable):

cd build
rm -r *   # restart configuration just in case
CMAKE_PREFIX_PATH=/data/data/com.termux/files/usr cmake .. -DLLAMA_CLBLAST=ON

I don't really know how Termux works though.

CLBlast_DIR is supposed to point at CLBlast's CMake files, on my system it is /usr/local/lib/cmake/CLBlast.

You can try to use CMAKE_PREFIX_PATH (environment variable):

cd build
rm -r *   # restart configuration just in case
CMAKE_PREFIX_PATH=/data/data/com.termux/files/usr cmake .. -DLLAMA_CLBLAST=ON

I don't really know how Termux works though.

I'll mess around with it tonight, and let you know how it goes tomorrow. Thanks for the cmake_prefix_path idea.

[SlyEcho]

OK, got Termux running in Docker.

First install some packages:

pkg update
pkg upgrade
apt install clang cmake cmake-curses-gui opencl-headers ocl-icd

Install CLBlast:

cd
git clone https://github.com/CNugteren/CLBlast.git
cd CLBlast
cmake -B build \
  -DBUILD_SHARED_LIBS=OFF \
  -DTUNERS=OFF \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=/data/data/com.termux/files/usr
cd build
make -j8
make install

Build llama.cpp:

cd
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp/
cmake -B build -DLLAMA_CLBLAST=ON
cd build
make -j8

This is fantasic, @SlyEcho . I genuinely appreciate.

I'm stuck during install CLBLAST. I run

cmake -B build \
  -DBUILD_SHARED_LIBS=OFF \
  -DTUNERS=OFF \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_PREFIX_PATH=/data/data/com.termux/files/usr

And receive

CMake Deprecation Warning at CMakeLists.txt:12 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Building CLBlast with OpenCL API (default)
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /data/data/com.termux/files/home/CLBlast/build

I tried continuing with,

cd build
make -j8

And am stuck on:

[100%] Built target clblast
Install the project...
-- Install configuration: "Release"
CMake Error at cmake_install.cmake:46 (file):
  file cannot create directory: /usr/local/lib.  Maybe need administrative
  privileges.


make: *** [Makefile:100: install] Error 1

Now of course llama.cpp is saying CLBlast not found.

I'm confused as to what is exactly causing the problem that I can't make install.

It feels like we're very close though, so thanks again for coming this far!

Please let me know if there's anything I can do to force this make install.

[SlyEcho]

CMake Error at cmake_install.cmake:46 (file):
 file cannot create directory: /usr/local/lib.  Maybe need administrative
 privileges.

It is trying to install into /usr/local/lib,

I made a mistake, it should be CMAKE_INSTALL_PREFIX

You can reconfigure with:

cmake .. -DCMAKE_INSTALL_PREFIX=/data/data/com.termux/files/usr
make install

If that path is not allowed either you can install in some home folder and then point llama.cpp to it with the CMAKE_PREFIX_PATH environment variable.

CMake Error at cmake_install.cmake:46 (file):
 file cannot create directory: /usr/local/lib.  Maybe need administrative
 privileges.

It is trying to install into /usr/local/lib,

I made a mistake, it should be CMAKE_INSTALL_PREFIX

You can reconfigure with:

cmake .. -DCMAKE_INSTALL_PREFIX=/data/data/com.termux/files/usr
make install

If that path is not allowed either you can install in some home folder and then point llama.cpp to it with the CMAKE_PREFIX_PATH environment variable.

Thank you. This worked for me. I'm saving these posts for myself to learn from. Llama.cpp found clblast, and I'm able to build it.

Now, I'm getting an error running ./main, so I might reference it in a new issue, but here's the message,

source:1:2115: warning: implicit conversion from 'const __private int32_t' (aka 'const __private int') to 'float' may lose precision                              typedef char int8_t; typedef uchar uint8_t; typedef int int32_t; typedef uint uint32_t; struct __attribute__ ((packed)) block_q4_0 { half d; uint8_t qs[QK4_0 / 2]; }; struct __attribute__ ((packed)) block_q4_1 { half d; half m; uint8_t qs[QK4_1 / 2]; }; struct __attribute__ ((packed)) block_q5_0 { half d; uint32_t qh; uint8_t qs[QK5_0 / 2]; }; struct __attribute__ ((packed)) block_q5_1 { half d; half m; uint32_t qh; uint8_t qs[QK5_1 / 2]; }; struct __attribute__ ((packed)) block_q8_0 { half d; int8_t qs[QK8_0]; }; __kernel void convert_fp16_to_fp32(__global half* x, __global float* y) { const uint i = get_global_id(0); y[i] = vload_half(0, &x[i]); } void dequantize_q4_0(__global const struct block_q4_0* x, const int ib, const int iqs, float* v0, float* v1) { const float d = vload_half(0, &x[ib].d); const uint8_t vui = x[ib].qs[iqs]; const int8_t vi0 = vui & 0xF; const int8_t vi1 = vui >> 4; *v0 = (vi0 - 8)*d; *v1 = (vi1 - 8)*d; } void dequantize_q4_1(__global const struct block_q4_1* x, const int ib, const int iqs, float* v0, float* v1) { const float d = vload_half(0, &x[ib].d); const float m = vload_half(0, &x[ib].m); const uint8_t vui = x[ib].qs[iqs]; const int8_t vi0 = vui & 0xF; const int8_t vi1 = vui >> 4; *v0 = vi0*d + m; *v1 = vi1*d + m; } void dequantize_q5_0(__global const struct block_q5_0* x, const int ib, const int iqs, float* v0, float* v1) { const float d = vload_half(0, &x[ib].d); uint32_t qh = x[ib].qh; const uint8_t xh_0 = ((qh >> (iqs + 0)) << 4) & 0x10; const uint8_t xh_1 = ((qh >> (iqs + 12)) ) & 0x10; const int32_t x0 = ((x[ib].qs[iqs] & 0xf) | xh_0) - 16; const int32_t x1 = ((x[ib].qs[iqs] >> 4) | xh_1) - 16; *v0 = x0*d; *v1 = x1*d; } void dequantize_q5_1(__global const struct block_q5_1* x, const int ib, const int iqs, float* v0, float* v1) { const float d = vload_half(0, &x[ib].d); const float m = vload_half(0, &x[ib].m); uint32_t qh = x[ib].qh; const uint8_t xh_0 = ((qh >> (iqs + 0)) << 4) & 0x10; const uint8_t xh_1 = ((qh >> (iqs + 12)) ) & 0x10; const int32_t x0 = ((x[ib].qs[iqs] & 0xf) | xh_0); const int32_t x1 = ((x[ib].qs[iqs] >> 4) | xh_1); *v0 = x0*d + m; *v1 = x1*d + m; } void dequantize_q8_0(__global const struct block_q8_0* x, const int ib, const int iqs, float* v0, float* v1) { const float d = vload_half(0, &x[ib].d); const int8_t vi0 = x[ib].qs[iqs + 0]; const int8_t vi1 = x[ib].qs[iqs + 1]; *v0 = vi0*d; *v1 = vi1*d; } void convert_f16(__global half* x, const int ib, const int iqs, float* v0, float* v1){ *v0 = vload_half(0, &x[ib + 0]); *v1 = vload_half(0, &x[ib + 1]); }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         

To clarify, I am able to build, and run llama.cpp using cmake, but with clblast enabled then I'm getting this a huge error in ./main

Again, thanks to you for helping me get it compiled at all!

[SlyEcho]

The messages are impossible to read because the CL program doesn't have line breaks, but here are the errors:

source:2:35: error: 16-bit storage is not supported for SSBOs
source:2:35: error: 8-bit storage is not supported for SSBOs
source:3:43: error: 16-bit storage is not supported for SSBOs
source:3:43: error: 8-bit storage is not supported for SSBOs
source:4:35: error: 16-bit storage is not supported for SSBOs
source:4:35: error: 8-bit storage is not supported for SSBOs
source:5:43: error: 16-bit storage is not supported for SSBOs
source:5:43: error: 8-bit storage is not supported for SSBOs
source:6:35: error: scalar elements must be aligned to their size

It just seems like this device doesn't support llama.cpp, maybe it only works with float32 numbers?

@0cc4m, what do you think?

The messages are impossible to read because the CL program doesn't have line breaks, but here are the errors:

source:2:35: error: 16-bit storage is not supported for SSBOs
source:2:35: error: 8-bit storage is not supported for SSBOs
source:3:43: error: 16-bit storage is not supported for SSBOs
source:3:43: error: 8-bit storage is not supported for SSBOs
source:4:35: error: 16-bit storage is not supported for SSBOs
source:4:35: error: 8-bit storage is not supported for SSBOs
source:5:43: error: 16-bit storage is not supported for SSBOs
source:5:43: error: 8-bit storage is not supported for SSBOs
source:6:35: error: scalar elements must be aligned to their size

It just seems like this device doesn't support llama.cpp, maybe it only works with float32 numbers?

@0cc4m, what do you think?

Thanks for cleaning the error message.

I'm confused because I use llama.cpp everyday, so it's definitely supported. Openblas works as expected. Perhaps it's just clblast that isn't supported? Which is still odd because running clpeak shows,

clpeak                                                                    Driver version  : 3.0 CLVK on Vulkan v1.1.128 driver 2149539840 (Android)                                   Compute units   : 2                                   Clock frequency : 0 MHz
                                                          Global memory bandwidth (GBPS)                          float   : 21.86
      float2  : 24.10
      float4  : 19.43
      float8  : 10.23
      float16 : 8.94
                                                          Single-precision compute (GFLOPS)
      float   : 369.29
      float2  : 273.19
      float4  : 309.08                                      float8  : 507.69
      float16 : 523.76

    No half precision support! Skipped                
    No double precision support! Skipped

    Integer compute (GIOPS)                                 int   : 109.64
      int2  : 71.84
      int4  : 139.36
      int8  : 80.51                                         int16 : 78.88

    Integer compute Fast 24bit (GIOPS)
      int   : 108.55                                        int2  : 71.70
      int4  : 139.01
      int8  : 80.41
      int16 : 77.72                                   
    Transfer bandwidth (GBPS)
      enqueueWriteBuffer              : 8.22
      enqueueReadBuffer               : 1.04                enqueueWriteBuffer non-blocking : 8.67
      enqueueReadBuffer non-blocking  : 1.05
      enqueueMapBuffer(for read)      : 8992.81
        memcpy from mapped ptr        : 1.04                enqueueUnmap(after write)       : 58355.54
        memcpy to mapped ptr          : 8.60

    Kernel launch latency : 27.10 us