open source · GPU-native · dual backend

One TUI for
vLLM and llama.cpp.

Two engines. One dashboard. Zero config headaches.
Pick a profile in the TUI or shell — llmux runs the right engine for you.

$ git clone https://github.com/Bae-ChangHyun/llmux.git

GitHub Docs

Docker

vLLM

llama.cpp

NVIDIA Python

the problem

Two backends. Twice the toolchain.

Different tooling per engine

vLLM wants HF Transformer weights. llama.cpp wants GGUF. Different images, different flags, different compose files.

$ vllm serve — or — $ llama-server --model ...

Port / GPU clashes

Start llama.cpp on 8080, forget that vLLM was already on it. Container crash, logs panic, you dig for the offender.

$ docker ps | grep 8080

GGUF download drudgery

Find the file name, hf download, move to models/, edit compose, hope the path matches.

$ hf download ... && vim docker-compose.yaml

No unified benchmark

Want to compare a GPTQ vLLM run against a Q4_K_M GGUF run? Write the curl by hand, time it yourself, twice.

$ time curl /v1/chat/completions ...

how it works

Three steps, either engine.

Clone & configure

One .env.common holds your HF token and cache path. Both backends share it.

$ git clone ... && cat > .env.common

Launch llmux

Press n, pick vLLM or llama.cpp, type a model name. Profile + config auto-generated.

$ uv run llmux

Enter → Start

Cross-backend conflict gate checks ports and GPUs, then spins the container on the OpenAI-compatible API.

Enter → Start → Serving :8000 / :8080

demo

See it in action

llmux

terminal

Quick Start in your terminal

$ git clone https://github.com/Bae-ChangHyun/llmux.git
$ cd llmux
 
$ cat > .env.common << 'EOF'
HF_TOKEN=hf_your_token_here
HF_CACHE_PATH=/home/user/.cache/huggingface
EOF
 
$ uv run llmux
 
✔ TUI launched — press n for new profile
✔ Select backend: vLLM / llama.cpp
✔ Enter model: Qwen/Qwen3-30B-A3B
⚠ conflict gate: no overlap on port 8000 / GPU 0
✔ Container started — serving at http://localhost:8000/v1

before & after

From two toolchains to one keyboard

two separate projects

Switch enginesleave one TUI, open another

Port conflictscrash, then debug

GGUF setupmanual hf download

Benchmarkcurl + time, per engine

Memory checkguess per engine

Mental overheaddouble

with llmux

Switch enginesone dashboard

Port conflictscross-backend gate

GGUF setupauto pull on start

Benchmarkbuilt-in for both

Memory checkhf-mem, one keystroke

Mental overheadsingle

features

Everything you need, nothing you don't

Unified TUI

Every vLLM and llama.cpp profile shows up in the same list. Start, stop, logs, edit — all keyboard-driven.

Cross-backend conflict gate

Catches port + GPU collisions between vLLM and llama.cpp before docker is even touched.

Live throughput & benchmark

A live tok/s column on the dashboard, polled from each container's /metrics (llmux stats). Plus a warmup + median /v1/chat/completions bench for either engine.

GGUF auto-download

llama.cpp profiles ride llama-server's -hf / -hff to download GGUFs straight into the host HF cache on first start. No host-side hf CLI, no manual models/ wiring.

Memory Estimator

Press m, type a HF repo, get per-GPU fit estimates powered by hf-mem. Works for both backends.

GPU Monitor

Real-time GPU usage bars on the dashboard, auto-refresh every 5 seconds. No more nvidia-smi in a loop.

Headless CLI parity

Every TUI feature has a Typer CLI twin — llmux up, bench, profile quick-setup, image build-dev, system mem-estimate. Scripted from day one.

Per-profile image pinning

Pin a profile to any docker image — vllm-dev:<branch>, an official version, or a custom registry tag — via profile.image_tag. UI override beats the pin, the pin beats the default.

Unified dev builds

Build vLLM or llama.cpp from source with one command. Auto-detects your GPU compute capability (CUDA_DOCKER_ARCH / TORCH_CUDA_ARCH_LIST) so the image only ships what your card actually runs.

Config parameter toggles

Flip any config flag on or off without deleting it — a row switch in the TUI, or config edit --disable/--enable from the CLI. Disabled flags are parked as inert comments, and your own hand-written notes survive every edit.

Rename without rebuilding

Rename a profile or a config in place — R in the TUI, or profile rename / config rename headless. Profiles pointing at a renamed config are repointed automatically, and a rename is refused while the container is running rather than orphaning it.

what's next

Roadmap

Profile clone across backends +

llmux profile clone duplicates a profile within a backend today. What's still open is cross-backend cloning — turning a vLLM profile into its llama.cpp GGUF equivalent in one keystroke, to pair with the built-in OpenAI-compatible bench for apples-to-apples throughput/latency.

Batch operations · Bundle export/import · Quantization recommender +

Start/stop multiple profiles at once · ship a full profile + config bundle between machines · suggest Q4_K_M vs Q8_0 given target GPU and the built-in memory estimator.

faq

Common questions

How do I choose between vLLM and llama.cpp? +

Rule of thumb: vLLM for high-throughput serving of HF Transformer weights on big GPUs; llama.cpp for GGUF on smaller / offloaded setups, CPU+GPU mixed, or exotic quantizations (Q4_K_M, Q8_0). llmux lets you keep profiles for both and benchmark them side by side.

Can I run a vLLM profile and a llama.cpp profile at the same time? +

Yes — as long as they don't share a port or GPU. The cross-backend conflict gate warns you pre-start if they do. Each profile declares its own gpu_id and port in profiles.yaml.

What about LoRA adapters? +

Supported for vLLM profiles (multi-adapter loading via enable_lora: true in profiles.yaml + LORA_BASE_PATH in .env.common). llama.cpp LoRA support is on the roadmap.

Is the llama.cpp web UI served too? +

No. llmux injects --no-webui unconditionally. Only the OpenAI-compatible HTTP API is exposed, so both engines look identical to clients.

What are the requirements? +

Docker with NVIDIA Container Toolkit, Python 3.10+, and an NVIDIA GPU. uv is recommended for the TUI's venv but pip works too.

One TUI for vLLM and llama.cpp.