--- title: "The Private AI Stack — Whitepaper" description: "How the whole agentic stack — reasoning, retrieval, embeddings, speech and document generation — runs on the user's own device by default, and why." url: "https://royalsoftworks.com/resources/private-first-architecture/" source: "https://royalsoftworks.com" format: "markdown" note: "Markdown rendering of the HTML page at `url`. Same content, same canonical URL." --- [All resources](https://royalsoftworks.com/resources/) # The Private AI Stack ## A Local-First Architecture for Capable, Governable AI **A Royal Softworks whitepaper · AssistantGeneral** --- ## Abstract Most AI products are thin clients for a vendor's cloud: every prompt, document, and piece of context is transmitted off the user's machine to be processed. That architecture is incompatible with the work of the organizations that most need AI — those bound by privilege, regulation, classification, or simple competitive secrecy. The usual "private" alternative, a local chat model, solves the data problem by giving up almost all capability. AssistantGeneral takes a third path: run the **entire agentic stack** — reasoning, retrieval, embeddings, speech, and document generation — **on the user's own device by default**, while leaving the option to attach any of eleven cloud models when the user (or their administrator's policy) chooses. This paper describes how that architecture works and why "private _and_ capable" is a structural property of the system rather than a configuration toggle. --- ## 1\. The trade-off everyone else accepts The market offers two shapes of AI tool: - **Cloud assistants** are maximally capable but exfiltrate data by design. Enterprise contracts add legal assurances, but the data still leaves the device and is processed on infrastructure the customer does not control. - **Local chat UIs** keep data on the device but stop at conversation. They cannot safely read a file, search a private corpus with any rigor, query a database, generate a formatted document, or run a scheduled task. Buyers are told to pick a corner: powerful-but-leaky, or private-but-inert. AssistantGeneral is built to refuse the choice. --- ## 2\. Design principle: local by default, cloud by consent Every subsystem that _can_ run on-device _does_ run on-device by default: | Capability | Default (on-device) | Optional | | --- | --- | --- | | Language model | Local llama.cpp server | Attach any of 10 cloud providers | | Embeddings | Local ONNX models + cache | — | | Retrieval / RAG | Local LanceDB + reranker | Attach remote Chroma/Qdrant if desired | | Speech-to-text | Local Whisper | — | | Text-to-speech | Local Piper / Supertonic | — | | Document generation | Sandboxed Pyodide (WASM) | — | | Governance plane | Customer-hosted admin server | — | The network is an _enhancement_, not a _dependency_. With no connection at all, the core product — chat, retrieval over your documents, dictation, document generation, and the agent loop — continues to function. --- ## 3\. The agent loop AssistantGeneral is not a single prompt-and-response call. It is a multi-node agent graph in which each stage is a discrete, individually tested component: 1. **Router** selects the appropriate specialist persona for the request. 2. **Planner** decomposes the task into a plan — and on follow-up turns plans only the _remaining_ work rather than restarting from scratch. 3. **Retrieval (RAG)** gathers relevant evidence from the local knowledge base before the model answers. 4. **Tool executor** runs model-requested tools, streams their progress inline in the chat, and races every tool against a hard timeout so that a single hung tool can never freeze the session. 5. **Groundedness gate** (see §5) verifies the draft's factual claims against the evidence actually gathered, and re-retrieves targeted evidence for any unsupported claim. 6. **Reviewer** confirms the result satisfies the original request before finishing. 7. **Safety breaker** enforces bounded failure, timeout, and wall-clock limits, ending a runaway loop with an explanation instead of silence. Across turns, a **durable task record** (goal, plan, artifacts, log, open issues) persists so that long, multi-session jobs do not lose the thread. Oversized context is compressed via summarization while preserving the system persona and the latest instruction verbatim. --- ## 4\. Retrieval: a real hybrid pipeline, entirely local "Chat with your documents" is only as good as its retrieval. AssistantGeneral's retriever is a full hybrid pipeline that runs on the device: ``` query → query planning (HyDE / multi-query expansion to widen the candidate net) → embed (local ONNX) → LanceDB hybrid search: dense vectors + keyword (BM25/FTS) → Reciprocal Rank Fusion of the two result sets → cross-encoder reranking (BGE) against the ORIGINAL query → recency weighting → MMR diversification (relevance without redundancy) → top-K passages + citation highlight spans ``` It returns the most relevant **passages**, not whole documents — better grounding and tighter citations. The corpus can additionally be **auto-clustered into communities** and expressed as a **knowledge graph**, surfaced as clickable links in the chat. Embeddings are cached so re-indexing is cheap, and external knowledge bases (local SQLite, a LanceDB folder, or remote Chroma/Qdrant) can be attached and merged with weighting. None of this requires a cloud round-trip to index or query private documents. --- ## 5\. Groundedness: the safety primitive for high-stakes answers A fluent answer that isn't tied to evidence is a liability in law, medicine, and finance. AssistantGeneral addresses this with a dedicated **groundedness gate** that runs after the model drafts an answer and before it ships: - It checks each **factual claim** in the draft against the evidence gathered this turn — the injected retrieval context and the tool outputs. - When claims are unsupported, it does not merely flag them. It **re-retrieves targeted evidence for exactly those claims** (a closed "re-RAG" loop), re-injects it, and sends the draft back to be supported with the new evidence, qualified, or removed. - The loop is **bounded** — a finished answer can be regenerated at most once across the groundedness and review gates together, so the system can never spin in a "keeps re-answering" loop. A mostly-good answer is annotated, not regenerated. The verdict runs on an internal model pass and never leaks into the answer; it surfaces as a visible process step, like the planner and reviewer. --- ## 6\. Acting safely: tools behind guards The agent's power comes from tools — filesystem, web, shell, data connectors, document generation, knowledge search. Power demands guardrails, which are built in: - **A hard filesystem security guard** makes system and credential locations (`/etc`, `~/.ssh`, `~/.aws`, `~/.kube`, Windows system directories, and the like) **unreadable** — not by convention, by enforcement. Reads outside the designated working folder require explicit, one-time confirmation. - **Confirmation gates** wrap dangerous operations (writes, deletes, shell execution). - **Read-only data access** — connectors expose discovery, search, fetch, schema, and **SELECT-only** query operations. The agent cannot write, modify, or delete remote data by design. - **Per-tool timeouts** ensure no tool can hang the session. --- ## 7\. Document generation in a sandbox To produce real `.xlsx`, `.docx`, and `.pptx` files — not Markdown approximations — the agent writes Python that executes in a **sandboxed Pyodide (WebAssembly) worker** with openpyxl, XlsxWriter, python-docx, python-pptx, NumPy, pandas, and matplotlib. The code runs in WASM isolation, on the device, with no network. Documents are built in **layers** and repaired **surgically** (line-level edits rather than full rewrites); spreadsheet math is validated; and the user's **original file is never overwritten** — edits are delivered as a separate copy. --- ## 8\. Model flexibility without lock-in A single provider registry resolves any of eleven backends — local llama.cpp (default), Anthropic, OpenAI, xAI, OpenRouter, Groq, Mistral, DeepSeek, Together, Google Gemini, or any custom OpenAI-compatible endpoint — with per-provider reasoning toggles and custom headers. The user can **switch models mid-conversation** without losing their knowledge base, and adding a new OpenAI-compatible provider is a single registry entry. The product benefits from cloud model progress without depending on any one vendor. --- ## 9\. Interoperability: MCP both directions AssistantGeneral is a first-class participant in the Model Context Protocol: it is both an MCP **client** (connect any external MCP server to add tools) and ships a built-in MCP **server** (expose its own capabilities to other MCP-aware applications). The private agent is part of the open agent ecosystem rather than a walled garden. --- ## 10\. Why this is an architecture, not a feature The reason a cloud-native competitor cannot simply add a "private mode" is that privacy here is a property of _every layer simultaneously_: the inference path, the embedding path, the retrieval store, the speech engines, the document sandbox, and the governance plane are all local. A product designed to process data in the vendor's cloud would have to be rebuilt to move all of that onto the device. Conversely, a local chat tool would have to build the agent loop, the hybrid retriever, the document engine, the connectors, and the governance plane to reach parity. "Private _and_ capable" is therefore not a corner either incumbent can step into cheaply. It is the corner AssistantGeneral was built to occupy. --- ## 11\. Summary AssistantGeneral demonstrates that the privacy/capability trade-off was never fundamental — only architectural. By running reasoning, retrieval, embeddings, speech, and document generation on the device, gating every action behind real guards, verifying answers against evidence, and keeping even the governance plane on the customer's own server, it delivers the full agentic stack to the organizations that the cloud-only incumbents structurally cannot serve. _To evaluate it on your own data, on your own machine, with the network unplugged — which is the only test that matters for this class of buyer — contact Royal Softworks._ ## Evaluate AssistantGeneral See the product this paper describes — a private, local-first AI agent with an enterprise control plane that runs on your own infrastructure. [Explore AssistantGeneral](https://royalsoftworks.com/products/assistant-general/) [Talk to us](https://royalsoftworks.com/contact/)