Problems when i try to use this inside the default python 3.10 docker container

[LLukas22]

When i try to install and use this package via a requirements file in the default 3.10 python container i get the following error when i try to import the module:
Failed to load shared library '/usr/local/lib/python3.10/site-packages/llama_cpp/libllama.so': /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version 'GLIBCXX_3.4.29' not found (required by /usr/local/lib/python3.10/site-packages/llama_cpp/libllama.so)

Am i doing something wrong? Or am i just missing some dependencies?

[abetlen]

Do you mind posting the Dockerfile? I haven't tried to use this inside a container but I'l investigate.

[LLukas22]

Alright i cant send you the Dockerfile, but i created a toy-example with your own server.

Dockerfile:

FROM python:3.10

#install
RUN pip3 install llama-cpp-python[server]

#Expose the ports
EXPOSE 8000

# Make a Dir for the models to be mounted into
RUN mkdir -p /var/lib/models

#Set environment 
ENV MODEL "/var/lib/models/ggjt-model.bin"

#Start server
CMD ["python3","-m","llama_cpp.server"]

I then started the container via this docker-compose file:

version: "3.9"
services:
 
  api:
    build:
      context: .
      dockerfile: Dockerfile
    image: llama-server
    container_name: llama-server

    ports:
      - "8000:8000"

    volumes:
      - [MODEL_DIR]:/var/lib/models

When running this the 'GLIBCXX_3.4.29' error is gone, but the model loads for a very long time and the container gets OOM killed sometimes. I'm running on 64GB so that's very strange. The other thing i noticed is that the container fails to bind the address but that's just docker IPv4 shenanigans. I will try to use llama-cpp-python[server] as a dependency in my other project and see if it will get rid of the 'GLIBCXX_3.4.29' error.

Oh and just for completness i was using this model.

[abetlen]

So the GLIBC error is fixed now? I would also make sure the OOM issue isn't some Docker default limit.

[LLukas22]

Thats the strange thing the dockerfile listed above works without any problems. But if im trying to run my Dockerfile im getting the "GLIBC" error.

This is my Dockerfile:

FROM python:3.10

RUN apt-get update -y
RUN apt-get install -y python3-pip graphviz-dev git gcc-4.9 
RUN apt-get upgrade -y libstdc++6 

#Expose the ports
EXPOSE 8001

# Copy Files
RUN mkdir -p /app
ADD ./ /app/
WORKDIR /app

# Install Requirements
RUN --mount=type=cache,target=/root/.cache/pip pip3 install -r ./api/requirements-cpu.txt

RUN --mount=type=cache,target=/root/.cache/pip pip3 install git+https://github.com/huggingface/transformers@a17841ac4945631e4e13c072fa2a329b98ebb8b6 

ENV PYTHONPATH "${PYTHONPATH}:/app"

#Build cache dir for transformers
ENV HF_HOME "/huggingface/cache"

CMD ["python3", "/app/api/main.py"]

and its using the following requirements file:

protobuf==3.20.1
huggingface_hub
llama-cpp-python[server]
openai
pynvml
farm-haystack

#api dependencies
psutil
fastapi
uvicorn
dependency_injector

My guess is one of the dependencies in the requirements does some weird stuff, but i actually have no clue where to start looking.

[jmtatsch]

I run within a Ubuntu container which works.
https://github.com/mkellerman/gpt4all-ui/ runs a 3.11 container and it works so I would guess the issue is not with llama-cpp-python but with your concrete dockerfile/requirements.

[jmtatsch]

Took the opportunity to shrink my own dockerfile:

FROM python:3.10

COPY .devops/requirements.txt requirements.txt
RUN pip install -r requirements.txt && rm -rf requirements.txt

ENTRYPOINT [ "python3", "-m", "llama_cpp.server" ]

works without a hitch with requirements.txt

llama-cpp-python
uvicorn
fastapi
sse_starlette
typing_extensions

[LLukas22]

I also played around a bit more and i couldnt get the container working, and i dont know why. All the other containers i build with llama-cpp-python work without any problems. I will probably host a seperate llama-cpp-python container and then use it via its REST api.

@abetlen Could we maybe get an official prebuild docker container for the REST server?

[abetlen]

Yeah if someone wants to open a PR I can test / review it when I have some time.

[LLukas22]

I could try later. But i guess i noticed another problem i had, when building my containers via github-actions and then trying to run them locally the containers exit with exitcode 132 hinting at an unsupported CPU instruction-set. If build and run locally they work without any problems.

My guess the is the llama.cpp dependencies are resolved in the build step of the container with the cpu feature flags of the github-actions-runner. Can i define specific feature flags used for the llama.cpp compilation at setup time or should i look into first building llama.cpp manually and then setting it via the environment?

[abetlen]

@LLukas22 that's definitely it, should check what llama.cpp does for this with their docker containers.

[LLukas22]

@abetlen Yeah its pretty weird, i played around a bit but can't get it to work even if i use QEMU to force a linux/amd64 platform while building the image on an github-actions-runner. And i'm actually really confused what instruction set is missing as i'm using relatively new processors (AMD Zen 2 & 3) so avx and avx-2 are definitely there.

But i'm also no docker pro, so maybe i'm just missing something.

[abetlen]

@LLukas22 is QEMU sufficient to emulate processor specific instruction sets? I'd imagine it's because of a mismatch between processor architectures in the github runner and your local machine.

[LLukas22]

@abetlen Theoretically QEMU should be able to emulate any cpu features, but i dont know how it is implemented in docker build.
They probably create a vm with the cpu flags host or max which will use all features of the host system. Then the problem with the architecture mismatch happens when i try to run it locally. But i'm still a bit confused thought nearly all x64 cpus released in the last <5 years implement the same instructionsets 🤔

[abetlen]

@LLukas22 https://github.com/ggerganov/llama.cpp/blob/master/.github/workflows/docker.yml describes how they build the llama.cpp docker image, not sure if you've checked that out