Run Bub with a local llama.cpp model

Connect the Bub application to a local llama.cpp server. This guide starts a GGUF Gemma model, checks its OpenAI-compatible endpoint, and sets the app’s model connection.

The example uses ggml-org/gemma-4-E2B-it-GGUF, a GGUF build of Google’s Gemma 4 E2B instruction-tuned model. Google’s Gemma 4 overview describes E2B and E4B as efficient models for mobile and edge devices, and the Gemma 4 model card documents capabilities, limits, and responsible-use considerations.

Before you begin

You need:

• Bub installed and runnable with bub --help.
• Docker installed.
• A GGUF model file under ~/.cache/llama.cpp/.
• Enough system memory for the quantization you choose. The Q8 Gemma 4 E2B GGUF file is about 5 GB on disk; runtime memory also depends on context size, batching, and GPU offload.

This tutorial uses these file names:

~/.cache/llama.cpp/ggml-org_gemma-4-E2B-it-GGUF_gemma-4-E2B-it-Q8_0.gguf
~/.cache/llama.cpp/ggml-org_gemma-4-E2B-it-GGUF_mmproj-gemma-4-E2B-it-Q8_0.gguf

If your files use different names, update the -m and --mmproj paths in the Docker command.

1. Start the local server

Set an API key for the local server:

export LLAMA_API_KEY="${LLAMA_API_KEY:-test}"

Start llama-server:

sudo docker run --rm -it \
  --security-opt label=disable \
  -p 127.0.0.1:8080:8080 \
  -v "$HOME/.cache/llama.cpp:/root/.cache/llama.cpp:ro" \
  ghcr.io/ggml-org/llama.cpp:full \
  --server \
  --host 0.0.0.0 \
  --port 8080 \
  --api-key "$LLAMA_API_KEY" \
  -m /root/.cache/llama.cpp/ggml-org_gemma-4-E2B-it-GGUF_gemma-4-E2B-it-Q8_0.gguf \
  --mmproj /root/.cache/llama.cpp/ggml-org_gemma-4-E2B-it-GGUF_mmproj-gemma-4-E2B-it-Q8_0.gguf

The Docker port is bound to 127.0.0.1 so the server is available only from the local machine. Change the port binding only if you intentionally want another machine to reach it.

On SELinux systems, --security-opt label=disable avoids bind-mount permission failures when the container reads model files from ~/.cache/llama.cpp. If you only need text input, remove the --mmproj line.

If Docker prints no ROCm-capable device is detected, the container can still fall back to CPU inference. That is enough to test the integration, but responses will be slower.

2. Test the OpenAI-compatible API

In another terminal, send a small chat request:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-4-E2B-it",
    "messages": [
      {"role": "user", "content": "hello"}
    ]
  }'

A working server returns a chat.completion JSON object with an assistant message.

3. Configure Bub

Point Bub at the local server:

export BUB_API_BASE="http://localhost:8080/v1"
export BUB_API_KEY="$LLAMA_API_KEY"
export BUB_MODEL="openai:gemma-4-E2B-it"

Run one Bub turn:

bub run "Reply with one short sentence: hello from a local model."

Bub now uses the local OpenAI-compatible endpoint for model calls. The turn pipeline, channels, tools, and tapes are unchanged.

~/bubbuild/bub$ bub run "Reply with one short sentence: hello from a local model."
2026-05-19 01:32:40.601 | INFO     | bub.builtin.agent:_run_tools_with_auto_handoff:271 - loop.step step=1 tape=becda04eb9f7369c__0b871d5e50e7c192 model=openai:gemma-4-E2B-it
2026-05-19 01:32:46.747 | INFO     | bub.builtin.store:fork:122 - Merged 7 entries into tape "becda04eb9f7369c__0b871d5e50e7c192"
[cli:local]
hello from a local model.

4. Check the model documentation before changing workloads

When you switch the model or quantization, check the upstream model documentation first:

• The Hugging Face GGUF card lists supported local runtimes and the available quantized files.
• The Gemma 4 model card documents input modalities, context windows, intended use, license, and risks.
• Evaluate the model on representative tasks, checking for incorrect, biased, or unsafe output.

Try a small local model when latency, privacy, cost, or offline access matter to your task. Check its answer quality on actual tasks before making it available to users.

Clean up

Stop the Docker container with Ctrl-C.

Unset the Bub overrides when you want to return to your previous provider:

unset BUB_API_BASE BUB_API_KEY BUB_MODEL LLAMA_API_KEY