Back to projects
2026 · AI & ML · Stable

llama-router — Local multi-model inference optimized with AI

Local inference system serving multiple LLMs on an AMD Radeon 680M iGPU with 24 GB shared RAM. Configuration is generated declaratively, validated automatically against hardware, with persistent KV cache and task-based tier selection. Demonstrates the optimal tuning of a complete inference stack using AI tools in the development cycle.

llama.cpp Vulkan Python Podman Jinja2 SQLite pytest GGUF API OpenAI-compatible

Problem

LLM inference on integrated hardware (iGPU without dedicated VRAM, system-shared RAM) requires very different optimization decisions than cloud or discrete GPU setups. I wanted a system that could serve several models simultaneously, pick the most suitable one per task, and reconfigure without restarting the container.

Solution

An OpenAI-compatible HTTP server with a multi-model router: one model in memory at a time (LRU swap on demand), persistent KV cache between swaps so changing tiers doesn't require re-processing the prompt, and a declarative configuration layer that separates deployment plans from per-model flags, validated against hardware before generating the file consumed by the server.

Key takeaways

  • Multiple models served from a single endpoint with transparent switching between them
  • Persistent KV cache: switching models and back doesn't pay the cost of re-processing the prompt
  • Configuration generated from declarative plans, not hand-edited
  • Automatic validation against the hardware profile before every change
  • 27+ GGUF models in the pool, 2 active deployment plans, automated integrity tests
  • Built iteratively with AI assistance: every tuning decision justified and reproducible

What it is

A local inference server that exposes multiple LLMs simultaneously under an OpenAI-compatible API. Clients request a model by name and the server handles loading it, maintaining its cache, and unloading it when another model needs the slot.

What sets it apart from more typical setups: only one model in memory at a time, with the KV cache serialized to disk between swaps. This allows dozens of models to be available without paying the cost of having them all loaded.

What it demonstrates technically

Three pieces that show how to build a custom inference system with AI assistance in the loop:

1. Stack decisions justified for atypical hardware

On an AMD iGPU without dedicated VRAM, the realistic options were CPU-only, Vulkan, or ROCm. Vulkan was chosen — the technical reason being that the iGPU has direct access to shared RAM (no VRAM-style copy), and keeping the model on “GPU” is what delivers usable throughput. This kind of decision (which backend, which offload flag, which quantization) is repeated for every model in the pool.

2. Declarative layer on top of a server with complex config

llama.cpp has dozens of interdependent flags per model. Instead of maintaining hand-edited INI files, the configuration lives in:

  • Plans (plans/*.json) — which model goes in each logical slot
  • Model profiles (presets/*/pool/*/config.json) — per-model flags
  • Hardware profile (hardware/profile.json) — system constraints

A script validates the combination against hardware and generates the file consumed by the server. Changing a model or its flags means editing a JSON and regenerating — not searching and replacing in an INI.

3. Validation before applying, not after

The flow for any configuration change includes a validation phase against hardware (does it fit in RAM? does the GGUF exist? are the flags valid?) and, where applicable, against a performance baseline (is throughput maintained? do the tool-calling tests still pass?). If the new config doesn’t pass, it is rejected before touching the container. If it passes, it is applied with automatic rollback on regression.

The router: what changes the experience

The component that justifies all the system complexity is in-memory model management:

  • A request arrives for model A → it loads, processes, its KV cache stays in RAM.
  • A request arrives for model B → A’s KV cache is serialized to disk, A is unloaded, B is loaded.
  • Another request comes in for A → A is restored from disk along with its KV cache from the snapshot. If the prompt matches the cache, it doesn’t re-process from scratch.

The result is that the system “feels” as if all models were always loaded, without paying the memory cost of actually having them loaded. In practice, switching between the small model (responds in ~0.5s) and the large model (responds in ~15s but reasons better) doesn’t penalize the user: the next exchange with either one finds the context hot.

Task-based optimization

Each model in the pool has a designated role — they aren’t loaded just because they’re interesting:

  • A small, fast model for tool-calling and short responses — the system’s “worker.”
  • A larger model with reasoning capability for planning and open-ended questions — the “strategist.”
  • Specialized models reserved for cases where the first two aren’t enough (code review, trading signals).

The client requests the model by name. The choice of which one to use is made by the agent consuming the service, based on which pipeline phase it is in (perception, execution, reflection).

How it was developed with AI

This project is an example of using AI tools in the development loop of a technical system:

  • Exploring the flag space — llama.cpp flags are documented but their interdependencies aren’t. AI assistance helped read community code, GitHub issues, and benchmark results to decide on reasonable combinations.
  • Generating the declarative layer — the “shape adapter” that translates between the internal JSON format and the INI consumed by the server is plumbing code, not research. Suitable for assisted generation.
  • Validation against hardware — writing invariants (does it fit in RAM? is the GGUF where it should be?) is faster with assistance, and automatic verification reduces regressions.
  • Iterative tuning — when a change degrades performance, the loop is: detect → revert → adjust the responsible flag → re-validate. AI accelerates the “adjust” and “explain why” steps, but the “detect” and “revert” steps are handled by the test system.

The result: 27+ models in the pool, 2 deployment plans, 48 integrity tests, and the confidence that a configuration change won’t break the system in production because there’s automatic validation before and rollback after.