2024年7月19日 (五) 02:56的版本

Build llama.cpp

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

Convert Hugging Face Model to GGUF

pip install -r requirements.txt
python convert_hf_to_gguf.py --help
#M1 MPS does not support bf16
python convert_hf_to_gguf.py ~/Documents/MODELS/Qwen2-0.5B --outfile ~/Documents/MODELS/qwen2-0.5b-fp16.gguf --outtype f16

Run the model

./llama-cli -m ~/Documents/MODELS/qwen2-0.5b-fp16.gguf -p "Hi, who are you?" -n 128

↑ https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md

[1] ttps://github.com/ggerganov/llama.cpp/blob/master/docs/build.md

[1]

@@ 第17行： / 第17行： @@
 </syntaxhighlight>
+=Run the model =
+<syntaxhighlight lang="bash">
+./llama-cli -m ~/Documents/MODELS/qwen2-0.5b-fp16.gguf -p "Hi, who are you?" -n 128
+</syntaxhighlight>
 [[Category:Deep Learning]]