Hugging Face Local Models

llama-cli

llama-server

Default Workflow

1. Search the Hub with

   apps=llama.cpp

   .
2. Open

   https://huggingface.co/<repo>?local-app=llama.cpp

   .
3. Prefer the exact HF local-app snippet and quant recommendation when it is visible.
4. Confirm exact

   .gguf

   filenames with

   https://huggingface.co/api/models/<repo>/tree/main?recursive=true

   .
5. Launch with

   llama-cli -hf <repo>:<QUANT>

   or

   llama-server -hf <repo>:<QUANT>

   .
6. Fall back to

   --hf-repo

   plus

   --hf-file

   when the repo uses custom file naming.
7. Convert from Transformers weights only if the repo does not already expose GGUF files.

Quick Start

Install llama.cpp

brew install llama.cpp
winget install llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make

Authenticate for gated repos

hf auth login

Search the Hub

https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending

Run directly from the Hub

llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Run an exact GGUF file

llama-server \
    --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \
    --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
    -c 4096

Convert only when no GGUF is available

hf download <repo-without-gguf> --local-dir ./model-src
python convert_hf_to_gguf.py ./model-src \
    --outfile model-f16.gguf \
    --outtype f16
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

Smoke test a local server

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "messages": [
      {"role": "user", "content": "Write a limerick about exception handling"}
    ]
  }'

Quant Choice

• Prefer the exact quant that HF marks as compatible on the

  ?local-app=llama.cpp

  page.
• Keep repo-native labels such as

  UD-Q4_K_M

  instead of normalizing them.
• Default to

  Q4_K_M

  unless the repo page or hardware profile suggests otherwise.
• Prefer

  Q5_K_M

  or

  Q6_K

  for code or technical workloads when memory allows.
• Consider

  Q3_K_M

  ,

  Q4_K_S

  , or repo-specific

  IQ

  /

  UD-*

  variants for tighter RAM or VRAM budgets.
• Treat

  mmproj-*.gguf

  files as projector weights, not the main checkpoint.

Load References

• Read hub-discovery.md for URL-first workflows, model search, tree API extraction, and command reconstruction.
• Read quantization.md for format tables, model scaling, quality tradeoffs, and

  imatrix

  .
• Read hardware.md for Metal, CUDA, ROCm, or CPU build and acceleration details.

Resources

• llama.cpp:

  https://github.com/ggml-org/llama.cpp

• Hugging Face GGUF + llama.cpp docs:

  https://huggingface.co/docs/hub/gguf-llamacpp

• Hugging Face Local Apps docs:

  https://huggingface.co/docs/hub/main/local-apps

• Hugging Face Local Agents docs:

  https://huggingface.co/docs/hub/agents-local

• GGUF converter Space:

  https://huggingface.co/spaces/ggml-org/gguf-my-repo