Virtual Assistant — Local AI assistant on iGPU, designed with V-Model and agent playbook

Motivation

OpenClaw recommends “using the most powerful model available,” while in practice models without tool-calling fine-tuning “get stuck in loops or fail to format JSON.” Hermes Agent demands 64K+ context tokens and cloud models. On an AMD Radeon 680M iGPU with 24 GB of shared RAM and no dedicated VRAM, those solutions don’t fit, run slowly, or are impractical for 24/7 use.

I wanted a customized, fluid assistant that could run on that hardware. And I wanted to learn how to build agents for real, not consume a framework.

That’s why, before touching any code:

• I adopted Karpathy’s LLM Wiki Pattern: project knowledge lives in a persistent wiki, not chat histories.
• I created the agent playbook in that wiki — glossary, fundamentals, context management, guardrails, validation, multi-agent systems, and multi-agent debate — as my own conceptual framework before implementing.
• I defined git rules (branch per feature, atomic commits, pre-merge validation, clean code rule) in the project’s agent rules file (AGENTS.md).
• I adopted the V-Model NASA as the development methodology for each iteration.

The result: a local-first AI assistant with three interfaces, Home Assistant integration, web search, persistent memory, and a reflection system that analyzes its own performance. A fluid assistant on modest hardware is achievable — and the pages that follow show how it was built and what latencies are achieved.

---

The approach: wiki first, code later

The difference between “using a framework” and “building an agent” lies in the order: decisions first (wiki), then implementation (code).

Knowledge layer	Where it lives	What it’s for
Agent playbook	wiki/concepts/playbook-*	Conceptual framework: agent loop, tool calling, context rot, guardrails, eval sets
V-Model NASA	wiki/concepts/v-model-nasa-desarrollo	Methodology: ConOps → Requirements → HLD → LLD → V&V per feature
Git workflow	Each project’s agent rules file	Operational rules: one branch per feature, validation before merge, clean code
Project designs	the project’s design pages	Work product of each V-Model iteration

This structure is self-reinforcing: when a feature enters a loop (a test fails twice in a row, an architectural decision wobbles), the wiki explains why and the V-Model forces a return to the Requirements phase before touching code again.

The git workflow is applied with discipline: 11 iterations merged to main since June 16 (v0.7 → v1.5.1), each in its own branch with a dedicated merge commit.

---

Architecture: 5 layers with a single contract

The 5 layers are the minimum that isolates I/O, agent logic, tools, memory, and LLM into independently testable boundaries. The Core has a deterministic dispatcher (3-level cascade: keywords → cross-encoder → embeddings) that resolves ~80% of traffic in <50ms without calling the LLM; the remaining ~20% goes through all 6 phases with a round loop and tool-calling.

Real status of each layer

• Interfaces — CLI in REPL, Telegram via raw httpx (30 tests), Voice with a custom adapter that speaks Wyoming to Parakeet STT and Piper TTS es_ES-davefx-medium.
• Tooling — ~16 tools organized into 5 intent domains:
  • Home Automation (Home Assistant): turn devices on/off, adjust values (lights, climate, blinds), query any entity’s state, list the device catalog, launch vacuum routines.
  • Web: general search and page reading, with result grounding to avoid hallucinations.
  • System: Podman container inspection and basic fleet diagnostics.
  • Information and Context: weather, upcoming events, date and time as temporal anchors, and two synthesis tools that combine multiple sources when the user asks an open-ended question like “what to do today.”
  • Cross-cutting Grounding: validates that every agent claim is backed by a real tool result, in <1ms without additional LLM.
• Memory — Working in-memory, Episodic persisted in JSONL, Semantic with routine and procedure detection, and a /memory consolidate command that batches patterns. All accessible via CLI /memory subcommands.
• LLM Gateway — llama-router with two active routing tiers: standard (Qwen3.5-2B) and capable (LFM2.5-8B-A1B MoE). The CLI supports --model capable for intentional routing.

---

Methodology: V-Model NASA + Loop Engineering

The V-Model is executed operationally as a chain of Generator-Verifier loops with a hard budget (Loop Engineering, Addy Osmani 2026). Each V phase has its budget: if the Verifier fails twice in a row, the issue is escalated to the human. If the budget runs out, same thing. Final acceptance is done by a human running the CLI in REPL — the LLM never validates acceptance.

In this project the V is realized as: 11 design pages spanning iterations v0.7 through v1.5.1, 800 pytest tests covering unit + integration + voice mocks, 30 evaluation scenarios executable via the eval runner, and the acceptance matrix updated with every merge. The reflection phase (RuntimeAnalyzer and the tool_error_rate and memory_hits_rate metrics) closes the loop: it detects when a feature degrades the system and proposes IMPs to the human.

---

Inference: llama-router on the iGPU

llama-router is the HTTP server of llama.cpp compiled with the Vulkan backend (the only real option for accelerating small models on AMD GPUs) and extended with the --models-preset option. Instead of exposing a single model at a time, it reads a models.ini file with sections and maps each section to a different GGUF. The agent HTTP request includes a model= parameter and the router serves the corresponding GGUF; switching tiers requires no restart.

# models.ini (extracto del perfil activo lfm25-8b-capable)
[standard]
m = /models/Qwen3.5-2B-Q4_K_M.gguf
ctx-size = 16384
n-gpu-layers = 99
temp = 0.3
top-p = 0.8

[capable]
m = /models/LFM2.5-8B-A1B-Q4_K_M.gguf     # MoE 8.3B / 1.5B activos
ctx-size = 16384
n-gpu-layers = 99
temp = 0.3
top-p = 0.9
reasoning-budget = 4096

Active tiers in the agent’s real routing:

Tier	Model	Size	Role in the agent
standard	Qwen3.5-2B (Q4_K_M)	2B params	Executor: tool calls, short answers, ~80% of traffic
capable	LFM2.5-8B-A1B (Q4_K_M, MoE)	8.3B / 1.5B active	Planner: reasoning, multi-step, web search, vague-query grounding

The [coding] and [finance] sections are defined in models.ini (with Qwen2.5-Coder-3B and Qwen3-4B-Instruct-Alpha respectively) but are not used in current routing: the agent only routes between standard and capable. Coding is reserved for a future iteration of self-improvement of the assistant itself (UX); finance is not a project need.

Predictive routing sends ~80% of requests to the standard tier and only escalates to capable when the Perception phase detects signals of complexity (multi-step, multi-entity, composite, ambiguous) or when the 3-level cascade cannot resolve with enough confidence.

---

Latency results (measured on AMD Radeon 680M Vulkan)

Tier	p50 (latency)	p95 (latency)	Speed	Typical use
standard	0.5s	1.5s	~50 tok/s	”turn on the light”, “what’s the temperature?”, short tool calls
capable	~15s	~60s	~25 tok/s (MoE)	“find movie showtimes”, “what to do today in Madrid”

Data measured on AMD Ryzen 5 6600U + Radeon 680M iGPU (Vulkan), 24 GB shared RAM, Podman container with --device /dev/dri. Active profile: lfm25-8b-capable.

Key point: the standard tier — used in 4 out of 5 interactions — responds in half a second in the typical case and in just over a second in the worst case (p95). This is the range where an assistant feels “fluid” in conversation, indistinguishable from a bot calling a cloud API.

To put the hardware in perspective: LFM2.5-8B-A1B on a DGX Spark (NVIDIA GB10, 128 GB unified, 273 GB/s) does ~53 tok/s; on this iGPU (Radeon 680M, ~100 GB/s shared RAM) it does ~25 tok/s. The difference is ~2x, explainable by the combination of higher memory bandwidth (~2.7x) plus the DGX’s computational advantage. It’s not 10x, as the price difference would suggest.

---

Conclusion: a fluid assistant on modest hardware is achievable

With the right hardware (Vulkan on AMD iGPU, small active MoE models, Q4_K_M quantization), a 5-layer architecture, a rigorous methodology (V-Model + Loop Engineering + playbook), and a tiered router — yes, it can be done. The system runs 24/7 on a home server drawing ~25W at idle and ~35W under load. The assistant’s electricity bill is that of a light bulb.

The keys that made the project viable:

1. Wiki first — the agent playbook in the wiki forced conceptual clarity before touching code.
2. V-Model per feature — each iteration has its design page (one per V-Model phase: ConOps, Requirements, HLD, LLD, V&V). No requirements, no implementation.
3. Tiered router — 80% of interactions use the small, fast model; the 20% that need reasoning scale to MoE.
4. Deterministic cascade — Perception resolves most intents without an LLM in <50ms. This changes the feel of fluidity more than anything else.
5. AI-free grounding — response validation (grounding) uses regex and domain rules, <1ms; no additional LLM competes for the GPU.
6. MCP-style tool registry — a tool is registered once and becomes available for CLI, Telegram, and Voice.
7. Self-correcting reflection — RuntimeAnalyzer detects degradation and proposes IMPs; persistent memory enables learning across sessions.

---

Current status and roadmap

The project is active and running in home production. What works today:

• ✅ 3 operational interfaces: CLI, Telegram, Voice

• ✅ ~16 tools grouped into 5 domains (home automation, web, system, information, grounding)

• ✅ 2 active LLM tiers in routing (standard, capable)

• ✅ Persistent memory (JSONL episodic + JSON semantic with routines/procedures)

• ✅ Full reflection (RuntimeLogger, RuntimeAnalyzer, metrics, /health, /sugerencias)

• ✅ Eval runner with 30 YAML scenarios (--eval)

• ✅ 800 pytest tests passing

• ⏳ coding tier for assistant self-improvement — not yet implemented, on the roadmap as a UX goal

• ⏳ finance tier — not a project need, remains in models.ini as a theoretical definition

The iteration roadmap (v0.7 → v1.5+) is documented in the project’s roadmap page. Upcoming work areas include exploring the coding tier for the agent’s own internal tasks (reviewing PRs, refactors) and continuing to reduce latencies in Perception and Context.

---

References

• V-Model (Wikipedia) — development methodology
• Loop Engineering (Addy Osmani, 2026) — Generator-Verifier practice with turn budget
• Model Context Protocol — tool manifests