Eval bug: microsoft/bitnet-b1.58-2B-4T-gguf

[celsowm]

Name and Version

PS C:\Users\celso> llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
version: 5052 (1be76e4)
built with MSVC 19.41.34120.0 for x64

Operating systems

Windows

GGML backends

CUDA

Hardware

RTX 3060 12GB

Models

https://huggingface.co/microsoft/bitnet-b1.58-2B-4T-gguf

Problem description & steps to reproduce

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
build: 5052 (1be76e4) with MSVC 19.41.34120.0 for x64
system info: n_threads = 6, n_threads_batch = 6, total_threads = 12

system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8081, http threads: 11
main: loading model
srv load_model: loading model '.\ggml-model-i2_s.gguf'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3060) - 11247 MiB free
gguf_init_from_file_impl: tensor 'blk.0.ffn_down.weight' of type 36 (TYPE_IQ4_NL_4_4 REMOVED, use IQ4_NL with runtime repacking) has 6912 elements per row, not a multiple of block size (0)
gguf_init_from_file_impl: failed to read tensor info
llama_model_load: error loading model: llama_model_loader: failed to load model from .\ggml-model-i2_s.gguf

First Bad Commit

No response

Relevant log output

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
build: 5052 (1be76e46) with MSVC 19.41.34120.0 for x64
system info: n_threads = 6, n_threads_batch = 6, total_threads = 12

system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8081, http threads: 11
main: loading model
srv    load_model: loading model '.\ggml-model-i2_s.gguf'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3060) - 11247 MiB free
gguf_init_from_file_impl: tensor 'blk.0.ffn_down.weight' of type 36 (TYPE_IQ4_NL_4_4 REMOVED, use IQ4_NL with runtime repacking) has 6912 elements per row, not a multiple of block size (0)
gguf_init_from_file_impl: failed to read tensor info
llama_model_load: error loading model: llama_model_loader: failed to load model from .\ggml-model-i2_s.gguf

[BradHutchings]

convert_hf_to_gguf.py doesn't like it either. Looks like support needs to be added.

INFO:hf-to-gguf:Loading model: bitnet-b1.58-2B-4T
ERROR:hf-to-gguf:Model BitNetForCausalLM is not supported

[chenrq2005]

I got the same error on a 4090 nvidia GPU

[qjtdsqqm94akkyysgjdqo3hx1jn6l17]

Glad to see I wasn't the only moron who attempted to run this model in llama.cpp instead of microsoft's custom version!

That aside, I'd like to divert the attention to this line on the error message:

gguf_init_from_file_impl: tensor 'blk.0.ffn_down.weight' of type 36 (TYPE_IQ4_NL_4_4 REMOVED, use IQ4_NL with runtime repacking) has 6912 elements per row, not a multiple of block size (0)

I feel like it's just Microsoft using a nonstandard tensor encoding. Note how most of the the model tensors show up as unknown Precision in hf gguf explorer.

After a bit of poking around in the repo, it seems that llama.cpp don't have support for the I2_S tensor type mentioned in the model name (ggml-model-i2_s)—idk, see this pr, more specifically, this comment. The OP also included a link to a commit with I2_S support for anyone feeling confident enough to build & test it out.

[Manamama]

I was another moron, these weeks ago, welcome to the club.

Still, even in the newest gguf-dump: Version: 0.16.3 :

ValueError: np.uint32(36) is not a valid GGMLQuantizationType

which reads cryptic to the "uninitiated" to Microsoft's custom version.

[sanchomuzax]

Tip only💡
BitNet models are currently not compatible with the official llama.cpp, but they work with the ikawrakow/ik_llama.cpp fork, which supports the I2_S quantization. Alternatively, you can use Microsoft’s official bitnet.cpp framework for CPU inference. 🤔

[Manamama]

Thanks. Yes, trying: https://github.com/ikawrakow/ik_llama.cpp out of curiosity.
FYI, the Microsoft's version took me two weeks to figure out how to make it run in prooted Debian in Droid, see my comment in microsoft/BitNet#206 (comment) (which is still refusing to compile in standard Termux for some weird reason), so I thought that the mainline lllama.cpp would handle it by now, but #12997 (comment) etc.

[aezendc]

Thanks. Yes, trying: https://github.com/ikawrakow/ik_llama.cpp out of curiosity. FYI, the Microsoft's version took me two weeks to figure out how to make it run in prooted Debian in Droid, see my comment in microsoft/BitNet#206 (comment) (which is still refusing to compile in standard Termux for some weird reason), so I thought that the mainline lllama.cpp would handle it by now, but #12997 (comment) etc.

what branch did you use?