The Private AI Stack

A Local-First Architecture for Capable, Governable AI

A Royal Softworks whitepaper · AssistantGeneral

---

Abstract

Most AI products are thin clients for a vendor's cloud: every prompt, document, and piece of context is transmitted off the user's machine to be processed. That architecture is incompatible with the work of the organizations that most need AI — those bound by privilege, regulation, classification, or simple competitive secrecy. The usual "private" alternative, a local chat model, solves the data problem by giving up almost all capability.

AssistantGeneral takes a third path: run the entire agentic stack — reasoning, retrieval, embeddings, speech, and document generation — on the user's own device by default, while leaving the option to attach any of eleven cloud models when the user (or their administrator's policy) chooses. This paper describes how that architecture works and why "private and capable" is a structural property of the system rather than a configuration toggle.

---

1. The trade-off everyone else accepts

The market offers two shapes of AI tool:

• Cloud assistants are maximally capable but exfiltrate data by design. Enterprise contracts add legal assurances, but the data still leaves the device and is processed on infrastructure the customer does not control.
• Local chat UIs keep data on the device but stop at conversation. They cannot safely read a file, search a private corpus with any rigor, query a database, generate a formatted document, or run a scheduled task.

Buyers are told to pick a corner: powerful-but-leaky, or private-but-inert. AssistantGeneral is built to refuse the choice.

---

2. Design principle: local by default, cloud by consent

Every subsystem that can run on-device does run on-device by default:

Capability	Default (on-device)	Optional
Language model	Local llama.cpp server	Attach any of 10 cloud providers
Embeddings	Local ONNX models + cache	—
Retrieval / RAG	Local LanceDB + reranker	Attach remote Chroma/Qdrant if desired
Speech-to-text	Local Whisper	—
Text-to-speech	Local Piper / Supertonic	—
Document generation	Sandboxed Pyodide (WASM)	—
Governance plane	Customer-hosted admin server	—

The network is an enhancement, not a dependency. With no connection at all, the core product — chat, retrieval over your documents, dictation, document generation, and the agent loop — continues to function.

---

3. The agent loop

AssistantGeneral is not a single prompt-and-response call. It is a multi-node agent graph in which each stage is a discrete, individually tested component:

1. Router selects the appropriate specialist persona for the request.
2. Planner decomposes the task into a plan — and on follow-up turns plans only the remaining work rather than restarting from scratch.
3. Retrieval (RAG) gathers relevant evidence from the local knowledge base before the model answers.
4. Tool executor runs model-requested tools, streams their progress inline in the chat, and races every tool against a hard timeout so that a single hung tool can never freeze the session.
5. Groundedness gate (see §5) verifies the draft's factual claims against the evidence actually gathered, and re-retrieves targeted evidence for any unsupported claim.
6. Reviewer confirms the result satisfies the original request before finishing.
7. Safety breaker enforces bounded failure, timeout, and wall-clock limits, ending a runaway loop with an explanation instead of silence.

Across turns, a durable task record (goal, plan, artifacts, log, open issues) persists so that long, multi-session jobs do not lose the thread. Oversized context is compressed via summarization while preserving the system persona and the latest instruction verbatim.

---

4. Retrieval: a real hybrid pipeline, entirely local

"Chat with your documents" is only as good as its retrieval. AssistantGeneral's retriever is a full hybrid pipeline that runs on the device:

query
  → query planning (HyDE / multi-query expansion to widen the candidate net)
  → embed (local ONNX)
  → LanceDB hybrid search: dense vectors + keyword (BM25/FTS)
  → Reciprocal Rank Fusion of the two result sets
  → cross-encoder reranking (BGE) against the ORIGINAL query
  → recency weighting
  → MMR diversification (relevance without redundancy)
  → top-K passages + citation highlight spans

It returns the most relevant passages, not whole documents — better grounding and tighter citations. The corpus can additionally be auto-clustered into communities and expressed as a knowledge graph, surfaced as clickable links in the chat. Embeddings are cached so re-indexing is cheap, and external knowledge bases (local SQLite, a LanceDB folder, or remote Chroma/Qdrant) can be attached and merged with weighting.

None of this requires a cloud round-trip to index or query private documents.

---

5. Groundedness: the safety primitive for high-stakes answers

A fluent answer that isn't tied to evidence is a liability in law, medicine, and finance. AssistantGeneral addresses this with a dedicated groundedness gate that runs after the model drafts an answer and before it ships:

• It checks each factual claim in the draft against the evidence gathered this turn — the injected retrieval context and the tool outputs.
• When claims are unsupported, it does not merely flag them. It re-retrieves targeted evidence for exactly those claims (a closed "re-RAG" loop), re-injects it, and sends the draft back to be supported with the new evidence, qualified, or removed.
• The loop is bounded — a finished answer can be regenerated at most once across the groundedness and review gates together, so the system can never spin in a "keeps re-answering" loop. A mostly-good answer is annotated, not regenerated.

The verdict runs on an internal model pass and never leaks into the answer; it surfaces as a visible process step, like the planner and reviewer.

---

6. Acting safely: tools behind guards

The agent's power comes from tools — filesystem, web, shell, data connectors, document generation, knowledge search. Power demands guardrails, which are built in:

• A hard filesystem security guard makes system and credential locations (/etc, ~/.ssh, ~/.aws, ~/.kube, Windows system directories, and the like) unreadable — not by convention, by enforcement. Reads outside the designated working folder require explicit, one-time confirmation.
• Confirmation gates wrap dangerous operations (writes, deletes, shell execution).
• Read-only data access — connectors expose discovery, search, fetch, schema, and SELECT-only query operations. The agent cannot write, modify, or delete remote data by design.
• Per-tool timeouts ensure no tool can hang the session.

---

7. Document generation in a sandbox

To produce real .xlsx, .docx, and .pptx files — not Markdown approximations — the agent writes Python that executes in a sandboxed Pyodide (WebAssembly) worker with openpyxl, XlsxWriter, python-docx, python-pptx, NumPy, pandas, and matplotlib. The code runs in WASM isolation, on the device, with no network. Documents are built in layers and repaired surgically (line-level edits rather than full rewrites); spreadsheet math is validated; and the user's original file is never overwritten — edits are delivered as a separate copy.

---

8. Model flexibility without lock-in

A single provider registry resolves any of eleven backends — local llama.cpp (default), Anthropic, OpenAI, xAI, OpenRouter, Groq, Mistral, DeepSeek, Together, Google Gemini, or any custom OpenAI-compatible endpoint — with per-provider reasoning toggles and custom headers. The user can switch models mid-conversation without losing their knowledge base, and adding a new OpenAI-compatible provider is a single registry entry. The product benefits from cloud model progress without depending on any one vendor.

---

9. Interoperability: MCP both directions

AssistantGeneral is a first-class participant in the Model Context Protocol: it is both an MCP client (connect any external MCP server to add tools) and ships a built-in MCP server (expose its own capabilities to other MCP-aware applications). The private agent is part of the open agent ecosystem rather than a walled garden.

---

10. Why this is an architecture, not a feature

The reason a cloud-native competitor cannot simply add a "private mode" is that privacy here is a property of every layer simultaneously: the inference path, the embedding path, the retrieval store, the speech engines, the document sandbox, and the governance plane are all local. A product designed to process data in the vendor's cloud would have to be rebuilt to move all of that onto the device. Conversely, a local chat tool would have to build the agent loop, the hybrid retriever, the document engine, the connectors, and the governance plane to reach parity.

"Private and capable" is therefore not a corner either incumbent can step into cheaply. It is the corner AssistantGeneral was built to occupy.

---

11. Summary

AssistantGeneral demonstrates that the privacy/capability trade-off was never fundamental — only architectural. By running reasoning, retrieval, embeddings, speech, and document generation on the device, gating every action behind real guards, verifying answers against evidence, and keeping even the governance plane on the customer's own server, it delivers the full agentic stack to the organizations that the cloud-only incumbents structurally cannot serve.

To evaluate it on your own data, on your own machine, with the network unplugged — which is the only test that matters for this class of buyer — contact Royal Softworks.