wiki-brain — Second brain with LLM Wiki Pattern and periodic ingester

Why

Without a persistent wiki, two problems accumulate in any project of significant scale:

1. Forgotten decisions — why this architecture was chosen, what alternatives were ruled out, what tradeoffs were accepted. They live in chat histories that disappear when the conversation closes.
2. Stale knowledge — papers, posts, discussions. The LLM responds with what it knew at the end of its training cutoff; new knowledge the human finds stays in their head or in browser tabs.

Wiki-brain tackles both. The first, with Karpathy’s LLM Wiki Pattern: the LLM writes to a persistent vault, the human reads and corrects, and decisions survive across sessions. The second, with a periodic ingester that brings new knowledge into the vault automatically.

Why the results are more accurate

The typical problem with “asking an LLM about a technical topic” is that the LLM responds with plausibility, not truth. If a paper was published two months ago, the LLM doesn’t know it. If the documentation changed, it never heard. The answer “looks good” but can be wrong in the detail that matters most.

Wiki-brain tackles this from two angles simultaneously:

1. The context always reflects the current state of the vault. When an AI assistant calls query_wiki("how do you do grounding in the agent?"), the server searches the vault’s FTS5 index (SQLite’s native full-text search), retrieves the relevant pages (with [[wikilinks]] pointing to related pages), and passes them to the LLM as context. The LLM responds with citations — the answer includes the exact wikilink where each claim came from. If the page says “grounding without LLM in <1ms”, the answer says “grounding without LLM in <1ms ([[concepts/agent-validation]])” and the human can navigate to the page and verify.

The result is very different from a bare LLM: the rate of “coherent hallucination” drops dramatically because the LLM isn’t improvising, it’s citing. And when it does improvise, the reader sees it (no citation) and can correct it.

2. The periodic ingester prevents obsolescence. The LLM-as-judge for the relevance classifier responds based on its own knowledge; but the vault contains the paper or analysis published last week. The LLM using wiki-brain to answer always has the up-to-date vault as context, so when the ingester brings in a new paper, the next response can already cite it.

In practice this means query_wiki’s answer matches reality: either the wiki page says the same thing as reality (and the LLM cites it correctly), or the page is out of date (and the ingester should update it soon, or the human should edit it). Either way, there’s a traceable source for every claim.

The periodic ingester

The ingester is the piece that keeps the vault alive without human intervention. It runs on a daily timer that fires around 06:00 (with a small randomized offset to avoid colliding with other system timers).

The scheduler is the module that runs the pipeline at the scheduled time. It can also be invoked manually to force a run or process a single source out of band.

Source registry: what gets ingested and when

The SourceRegistry is the component that decides what gets processed on each run. It uses hybrid storage:

• Canonical: a wiki page (the source registry) with structured YAML blocks organized by category (academic, corporate, medium, reddit).
• Cache: a SQLite sources table for fast programmatic queries (deadline, last_ingested, total_ingested).

The wiki page is the source of truth. The SQLite cache is rebuilt from the wiki on every init. This allows editing the source list directly from Obsidian, without touching the database.

Each source declares a schedule:

Schedule	”Due” threshold
daily	22 hours since last_ingested
weekly	6 days since last_ingested
monthly	27 days since last_ingested

A query to list due sources iterates them all, filters by enabled and the requested category, and returns those past the threshold (or all if forced). A source that has never been ingested counts as “due” immediately.

Pipeline: 5 stages per source

The IngestionPipeline runs the same 5-stage flow for each source the scheduler passes to it:

1. Fetch — the source-type-specific fetcher (arxiv / rss / medium / reddit) downloads the items. If there are no items, the source is marked as ingested and execution ends.
2. Dedup — each item carries a url_hash. It is checked against the SQLite ingested_urls table. Already-processed items are skipped. This guarantees that a paper appearing in two different sources is not processed twice.
3. Classify — the RelevanceClassifier receives the new items. It processes them in batches of 15 using an LLM via OpenRouter, but with decision caching: each url_hash is persisted in relevance_decisions with the routing decision. Retries of the same URL are idempotent and consume no tokens.
4. Dispatch — the IngestDispatcher executes the routing decision deterministically (no LLM):
   • create — creates a new page with the processed content.
   • merge — appends the item to an existing page (searched by slug/topic similarity).
   • stub — creates a minimal placeholder (low relevance, the human decides).
   • queue — saves to a queue for manual review.
5. Cleanup — rebuilds the FTS5 index (SQLite full-text search), marks the source as ingested, and writes to the ingestion_log in SQLite.

Each run returns a result with items_fetched, items_new, items_ingested, skipped_dedup, errors, and a routing_summary with counts per action. The scheduler aggregates results from all sources and, when configured to update the playbook, triggers a synthesis pass that consolidates recent findings into the agent’s playbook pages.

Supported source types

Category	source_type	Example
academic	arxiv	arXiv cs.CL, cs.AI, cs.LG
corporate	rss	Engineering blogs (Anthropic, OpenAI, Netflix, AWS)
medium	medium	Medium tags with quality filter
reddit	reddit	Technical subreddits (r/MachineLearning, r/LocalLLaMA)

The system is extensible: adding a new source is a YAML entry in the registry’s wiki page, with its category, source_type, config (URL, query, subreddit, etc.) and schedule.

MCP tools

The server exposes 9 tools via MCP (Model Context Protocol — the standard protocol that AI assistants use to call external tools) over stdio (standard process input/output, not network). Any compatible client (OpenCode, Claude Desktop, Cursor) can consume them directly:

Tool	Function
search_wiki	Search by keywords, tags, type, or status
read_page	Read a full page with metadata, wikilinks, and backlinks
list_pages	List pages with filters (type, tag, status, tier)
query_wiki	Natural language question → synthesized answer with [[wikilinks]] citations
compile_wiki	Cross-page synthesis of a complete topic
crystallize	Merge related pages into a “wisdom” page
lint_wiki	Semantic health check: contradictions, orphans, broken links
rebuild_index	Rebuild the FTS5 index (SQLite full-text search) from the vault’s pages
ingest_knowledge	Inject knowledge: search → enrich or create new page

The first four (search/read/list/query) are read operations used in every assistant session. compile_wiki and crystallize are consolidation tools used after pages accumulate. lint_wiki and rebuild_index are maintenance tasks. ingest_knowledge is the programmatic interface to the ingester for ad-hoc inserts during a session.

Result: from Karpathy’s base to a system that actually works in production

Karpathy’s LLM Wiki Pattern is the conceptual base: the LLM writes to a persistent vault, the human reads, and decisions survive across sessions. The pattern is elegant but stops at the threshold of “system”: it doesn’t say how the vault is structured, how knowledge enters it, or how an AI assistant actually accesses it.

Wiki-brain takes that base and turns it into a system that runs unattended. The engineering decisions that make it work:

• Hybrid storage with the wiki as canonical. The SourceRegistry is stored as a wiki page (the source registry) with structured YAML blocks. SQLite holds a programmatic cache that is rebuilt from the wiki on every init. Editing the source list is a matter of opening Obsidian and editing markdown — no database, no migration, no API call. The registry is a living document, not a config file.
• URL hash deduplication, not title-based. A paper appearing in two different sources (say, an arXiv listing and an RSS digest of the same week) is recognized as the same item via a cryptographic hash of its URL, not a fuzzy title match. The hash is the identifier; titles can vary, but URLs don’t lie.
• Decision cache for the relevance classifier. When the LLM-as-judge classifies an item, the routing decision (create / merge / stub / queue) is persisted by url_hash in relevance_decisions. A retry of the same URL never spends tokens. The ingester stays economically viable at scale — the marginal cost of reprocessing a source is zero.
• Deterministic dispatch, no LLM in the hot path. The IngestDispatcher executes the routing decision in pure code: create makes a new page, merge appends to a similar existing page, stub makes a placeholder, queue parks it for review. The LLM decides what to do; deterministic code decides how and where. The result is reliable, auditable, and idempotent.
• MCP as the integration surface. The 9 tools are not a UI or an API for humans — they are tools the AI assistant calls during normal work. The wiki is a dependency of the assistant, not a separate app the human remembers to open.

The result of these decisions is a system that runs in production with no human in the loop for ingestion. The vault grows every day. The assistant has a second source of knowledge — the curated, deduplicated, cited, up-to-date wiki — that it can query, cite, and reason over, alongside its training data. Wiki-brain is what turns the LLM Wiki Pattern from a useful idea into infrastructure.