How to build on windows?

[Zerogoki00]

Please give instructions. There is nothing in README but it says that it supports it

[cgcha]

At this point, there's support for CMake. The Python segments of the README should basically be the same. Once you install it, you can run

cmake -S . -B build/ -D CMAKE_BUILD_TYPE=Release

cmake --build build/ --config Release

I'm not actually sure if you need CMAKE_BUILD_TYPE=Release for the first command, but it ran for me.

Afterwards, the exe files should be in the build/Release folder, and you can call them in place of ./quantize and ./main

.\build\Release\quantize.exe .\models\7B\ggml-model-f16.bin .\models\7B\ggml-model-q4_0.bin 2

.\build\Release\llama.exe -m .\models\7B\ggml-model-q4_0.bin -t 8 -n 128

The current README points to a shell script for quantizing, but you can refer to an older version of the README for manual instructions.

[cgcha]

I usually run Linux, so I'm pretty unfamiliar with CMake, and there are probably better conventions for how to do this cleanly. I also tried everything in WSL and it seems to work fine.

[fgblanch]

Probably over engineered, I just got it working on windows by using gcc compiler included in Strawberry Perl and Make distributed with chocolatey:

• Install Strawberry Pearl: https://strawberryperl.com/
• Install Chocolatey: https://chocolatey.org/
• Install make distributed with chocolatey: choco install make

set CC=C:\Strawberry\c\bin\gcc.exe
set CXX=C:\Strawberry\c\bin\g++.exe 
make
quantize.exe .\models\7B\ggml-model-f16.bin q4_0.bin  2
main.exe -m q4_0.bin -t 8 -n 128 

[akshay-verma]

I will recommend using WSL2 on Windows, that's what I used and everything worked fine.
I followed the steps for running the model from here -
https://til.simonwillison.net/llms/llama-7b-m2

[YongeBai]

Probably over engineered, I just got it working on windows by using gcc compiler included in Strawberry Perl and Make distributed with chocolatey:

• Install Strawberry Pearl: https://strawberryperl.com/
• Install Chocolatey: https://chocolatey.org/
• Install make distributed with chocolatey: choco install make

set CC=C:\Strawberry\c\bin\gcc.exe
set CXX=C:\Strawberry\c\bin\g++.exe 
make
quantize.exe .\models\7B\ggml-model-f16.bin q4_0.bin  2
main.exe -m q4_0.bin -t 8 -n 128 

Tried these steps, ran into this error. Any ideas?

process_begin: CreateProcess(NULL, uname -s, ...) failed.
Makefile:2: pipe: No error
process_begin: CreateProcess(NULL, uname -p, ...) failed.
Makefile:6: pipe: No error
process_begin: CreateProcess(NULL, uname -m, ...) failed.
Makefile:10: pipe: No error
/usr/bin/bash: cc: command not found
I llama.cpp build info:
I UNAME_S:
I UNAME_P:
I UNAME_M:
I CFLAGS: -I. -O3 -DNDEBUG -std=c11 -fPIC -mfma -mf16c -mavx -mavx2
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC
I LDFLAGS:
I CC:
I CXX: g++.exe (i686-posix-dwarf, Built by strawberryperl.com project) 8.3.0

cc -I. -O3 -DNDEBUG -std=c11 -fPIC -mfma -mf16c -mavx -mavx2 -c ggml.c -o ggml.o
process_begin: CreateProcess(NULL, cc -I. -O3 -DNDEBUG -std=c11 -fPIC -mfma -mf16c -mavx -mavx2 -c ggml.c -o ggml.o, ...) failed.
make (e=2): The system cannot find the file specified.
make: *** [Makefile:186: ggml.o] Error 2

[fgblanch]

Probably over engineered, I just got it working on windows by using gcc compiler included in Strawberry Perl and Make distributed with chocolatey:

• Install Strawberry Pearl: https://strawberryperl.com/

• Install Chocolatey: https://chocolatey.org/

• Install make distributed with chocolatey: choco install make

set CC=C:\Strawberry\c\bin\gcc.exe

set CXX=C:\Strawberry\c\bin\g++.exe

make

quantize.exe .\models\7B\ggml-model-f16.bin q4_0.bin 2

main.exe -m q4_0.bin -t 8 -n 128

Tried these steps, ran into this error. Any ideas?

process_begin: CreateProcess(NULL, uname -s, ...) failed.

Makefile:2: pipe: No error

process_begin: CreateProcess(NULL, uname -p, ...) failed.

Makefile:6: pipe: No error

process_begin: CreateProcess(NULL, uname -m, ...) failed.

Makefile:10: pipe: No error

/usr/bin/bash: cc: command not found

I llama.cpp build info:

I UNAME_S:

I UNAME_P:

I UNAME_M:

I CFLAGS: -I. -O3 -DNDEBUG -std=c11 -fPIC -mfma -mf16c -mavx -mavx2

I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC

I LDFLAGS:

I CC:

I CXX: g++.exe (i686-posix-dwarf, Built by strawberryperl.com project) 8.3.0

cc -I. -O3 -DNDEBUG -std=c11 -fPIC -mfma -mf16c -mavx -mavx2 -c ggml.c -o ggml.o

process_begin: CreateProcess(NULL, cc -I. -O3 -DNDEBUG -std=c11 -fPIC -mfma -mf16c -mavx -mavx2 -c ggml.c -o ggml.o, ...) failed.

make (e=2): The system cannot find the file specified.

make: *** [Makefile:186: ggml.o] Error 2

It seems you forgot to set gcc as CC command. Try running:

set CC=C:\Strawberry\c\bin\gcc.exe

[kaliber91]

main:` prompt: 'The first man on the moon was'
main: number of tokens in prompt = 8
     1 -> ''
  1576 -> 'The'
   937 -> ' first'
   767 -> ' man'
   373 -> ' on'
   278 -> ' the'
 18786 -> ' moon'
   471 -> ' was'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000


The first man on the moon was a geologist, and he brought his hammer.
Inside Out is an amazing movie that will take you through all kinds of emotions in its 90 minute run time (and maybe even more during your afterthoughts). The film tells about RileyÔÇÖs journey when she moves from Minnesota to San Francisco for a new job opportunity and how her parents, boyfriend Oliver Tate (!) and friends help her cope with that.
The animation looks great as always in Pixar productions but even more importantly the characters feel believable ÔÇô if you would have asked me before I watched Inside

main: mem per token = 14565444 bytes
main:     load time =  1157.11 ms
main:   sample time =   114.25 ms
main:  predict time = 19469.45 ms / 144.22 ms per token
main:    total time = 21031.82 ms

It works great on Windows using the CMake. Though -t 16 is no faster than -t 8 Ryzen 9 5950x. I regenerated the prompt couple of times on 7B, and about half the time it gets it right.

[Christoph-Wagner]

The current README points to a shell script for quantizing, but you can refer to an older version of the README for manual instructions.

param([string]$modelPath, [switch]$removeF16) 

Get-ChildItem $modelPath -Filter ggml-model-f16.bin* | 
Foreach-Object {
    $newName = $_.FullName.Replace("f16","q4_0");
    Start-Process -FilePath ".\build\Release\quantize.exe" -ArgumentList $_.FullName, $newName, "2" -Wait 
    if ($removeF16) {
        Remove-Item $_.FullName
    }
}

Call it like this

.\quantize.ps1 -modelPath "C:\PathToModels\65B" or .\quantize.ps1 -modelPath "C:\PathToModels\65B" -removeF16

Just thought I’d share this quickly thrown together powershell script for the Windows version of quantize.sh

[akshay-verma]

@kaliber91 7B was terrible for me as well. 13B was a bit better.

[Reelix]

Solving some common issues people might come across on the latest version of Python when installing the requirements.

This is specifically here as installed Windows versions of Python have compatibility issues with the chosen packages.

python -m pip install numpy
pip3 install torch -f https://download.pytorch.org/whl/torch_stable.html (About a 3GB download)
pip install .\sentencepiece-0.1.97-cp311-cp311-win_amd64.whl

The sentencepiece-0.1.97-cp311-cp311-win_amd64.whl file is from here inside the wheelhouse folder.

[kassane]

If you're running WSL2, it requires the creation or modification of a .wslconfig file in your user folder.

%USERPROFILE%\.wslconfig:

[wsl2]
memory=12GB
processors=6
swap=4GB

My Setup

• RAM: 16GB DDR4
• CPU: Ryzen 7 7500G
• SSD: 480GB
• OS: Windows 11

---

Based on this configuration. I succeeded in making the model conversions. However, when running main it still slows down when reading the model and continuously consumes a lot of memory.

edit ---