Slow inference time running on CPU

[fozziethebeat]

System Info

I'm running the top listed model on a Linux x86 machine and I'm seeing extremely slow inference times and throughput when using CPU. Everything is fast and as expected when running on GPU.

On CPU, I'm running with

docker run  -p 5252:80 -v ./data:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cpu-0.2.2 --model-id BAAI/bge-large-en-v1.5

When sending traffic to it via locust, I'm consistently seeing response over 1 second.

I'm comparing this to a pretty low effort sentence-transformers server:

import uvicorn
from fastapi import FastAPI 
from pydantic import BaseModel
from typing import List
from sentence_transformers import SentenceTransformer
app = FastAPI()

class EmbedRequest(BaseModel): 
    inputs: str

class EmbedResponse(BaseModel):
    embedding: List[float]


@app.post("/embed")
def embed(request: EmbedRequest):
    print(request.inputs)
    embeddings = app.model.encode([request.inputs])
    return EmbedResponse(embedding=[])  # embeddings[0])


if __name__ == "__main__":
    app.model = SentenceTransformer("BAAI/bge-large-en-v1.5")
    uvicorn.run(app, host="localhost", port=5252, log_level="info")

This really basic server is getting average response times of about 450ms.

Am I using the CPU version correctly? Should it be significantly slower than inference with sentence-transformers for the same model?

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

See above

Expected behavior

Faster throughput on CPU than a naive sentence-transformers server using FastAPI.

[OlivierDehaene]

Do you happen to have a AMD CPU?

[fozziethebeat]

I do not. Here's the first bits of lscpu:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  24
  On-line CPU(s) list:   0-23
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz
    CPU family:          6
    Model:               85
    Thread(s) per core:  2
    Core(s) per socket:  12
    Socket(s):           1
    Stepping:            7
    CPU max MHz:         4800.0000
    CPU min MHz:         1200.0000
    BogoMIPS:            6999.82

[LLukas22]

Just adding that i saw the exact same behaviour, with the cpu only image. The problem even seams to get worse if i try to pass in a batch of inputs at once, i compared it against the python wrapped version of candle and the text-embeddings-inference took about 1 min for a batch of 32 inputs while a simple local candle embedding server took only a few seconds.

[OlivierDehaene]

@LLukas22 can you share more on this?
The bert CPU impl is almost exactly the same as the one in Candle Transformers. This might be only linked to the default float16 dtype that seem to be way slower than float32.

[LLukas22]

I tested TEI against both a f32 and a f16 model, f16 models seam to be a bit slower than their f32 counterpart but its not significant. TEI seams to get a lot slower compared to the local executed models with rising sequenze lengths, here is a plot from my "benchmark" script:

I tested my python wrapped bert example vs a TEI server that ran on the same device in a docker container. The default transformers implementation is still a lot faster than candle's cpu implementation, but TEI is nearly 10x slower for longer sequences. Here is the script i used: https://gist.github.com/LLukas22/7e4129288e2819c87e6cf4c4bbac939c

[OlivierDehaene]

Oh that's expected given your gist: TEI does not batch on CPU (yet). That's a different issue alltogether.
Here the main problem is that MKL's sgemm is slower than whatever PyTorch uses.

[LLukas22]

TEI does not batch on CPU (yet).
Well that certainly explains why it's a lot slower for batched inputs. I guess thats something that could easily be added later.

Regarding the gemm performance differences compared to pytorch, it's probably noteworthy that i didn't use MKL for my local tests. I probably have to look into how torch manages to achieve this inference speed.

Also tagging @LaurentMazare on this, since he probably knows a lot more about this.

[LaurentMazare]

Regarding the gemm performance differences compared to pytorch, it's probably noteworthy that i didn't use MKL for my local tests. I probably have to look into how torch manages to achieve this inference speed.

It would be very interesting to how the benchmark go when using mkl on the candle side - this usually make a large difference.
Also I would have thought that PyTorch is likely to use mkl behind the hood, you can check whether your version is compiled with mkl support by running torch.backends.mkl.is_available().

[LLukas22]

PyTorch is likely to use mkl behind the hood, you can check whether your version is compiled with mkl support by running torch.backends.mkl.is_available()

Pytorch seams to use mkl, i tried to compile the pyo3 wrapper with mkl support but the results are nearly identical (maybe a bit better). Could be that i configured mkl wrong or that the python wrapper introduces additional overhead.

[LaurentMazare]

We expose a utils.has_mkl on the pyo3 side if you want to check that mkl is enabled (and you can also check via ldd).
Note that it's pretty tricky to get it enabled, you need to add the following when compiling a binary and should probably do the same for the pyo3 bindings (which would also require adding a mkl feature in the cargo file).

#[cfg(feature = "mkl")]
extern crate intel_mkl_src;

Let me know if it doesn't work and I can give it a try (also feel free to make an issue on the candle repo for these bits).

[OlivierDehaene]

#35 helps on AMD CPUs (20% faster on average) but it shouldn't really make a diff on Intel ones besides making it clear to MKL that we want to use the latest instructions.

[LaurentMazare]

Sounds great! Do you have a sense of which part actually helps, mkl-dynamic/something else ? Also the 20% is compared to the standard mkl version or something else ?

[karan00713]

@fozziethebeat @OlivierDehaene @LaurentMazare hey ! i'm trying to use Sentence Transformer for embedding in the same way described in first comment's image
import uvicorn
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List
from sentence_transformers import SentenceTransformer
app = FastAPI()

class EmbedRequest(BaseModel):
inputs: str

class EmbedResponse(BaseModel):
embedding: List[float]

@app.post("/embed")
def embed(request: EmbedRequest):
print(request.inputs)
embeddings = app.model.encode([request.inputs])
return EmbedResponse(embedding=[]) # embeddings[0])

if name == "main":
app.model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
uvicorn.run(app, host="localhost", port=5000, log_level="info")

, but there is some memory leak happening, the memory gets increased after each request for embedding and not decreasing, i'm working on my local laptop with CPU , can you help with this ?

[Isaac4real]

I'm having exactly the same issue in CPU with batch documents. 3X slower than with sentence_transformers. I'm using a c5.2xlarge from AWS