Where Does That Prompt Go? What Happens When You Paste Sensitive Data Into a Cloud AI

This is a technical explainer, not legal advice. HIPAA determinations depend on your specific role, data, and vendor agreements — confirm with your compliance team or counsel.

A prompt box feels private the way a text message to yourself feels private. It isn't. Whatever you type into a cloud AI chat is sent, as plain request content, to that vendor's servers to be processed — the same basic mechanic as any other API call. That's not a flaw in any particular product; it's how a cloud model necessarily works, since the model itself runs on the vendor's hardware. The part worth being precise about is what that implies once the text you paste is something like a patient note, a client's case file, or unreleased financial data.

The HIPAA mechanics, specifically

Under HIPAA, if a vendor "creates, receives, maintains, or transmits" PHI (protected health information) on behalf of a covered entity, that vendor is a business associate, and a Business Associate Agreement (BAA) has to be in place before PHI can be shared with them at all (HHS.gov, Business Associates). This isn't a formality — a BAA obligates the vendor to specific safeguards and restrictions on how that data can be used.

The detail that catches people out: the free or consumer tier of a mainstream chat AI does not sign a BAA. Enterprise or API-tier offerings from some vendors do, but it's product- and vendor-specific, and it has to be the version you're actually using — not just something the vendor offers somewhere in their lineup. Pasting PHI into a consumer chat window without a BAA in place is a HIPAA violation regardless of how careful or well-intentioned the person doing the pasting is. And if that vendor's own infrastructure relies on a subcontractor — another cloud provider running underneath them — HIPAA's downstream-liability rule (45 CFR § 164.308(b)(4)) means a BAA is required at that layer too, which is exactly the kind of dependency that's invisible from a chat window.

This isn't unique to healthcare, either — it's the general shape of the problem for any regulated or simply sensitive text: legal privilege, trade secrets, an unreleased product roadmap. The mechanism differs by industry (privilege rules instead of a BAA, an NDA instead of 45 CFR), but the root fact is the same: the text left your machine and now lives on infrastructure you don't control, governed by an agreement you may or may not actually have.

What changes when inference runs on-device

If the model doing the reasoning is running on the same machine as the person typing, there's no vendor infrastructure in the loop for that request — the "did we get a BAA with the AI vendor" question doesn't come up, because no data transmission to an AI vendor occurred. This is the premise AssistantGeneral is built around: reasoning, retrieval, embeddings, speech-to-text and text-to-speech all run on the user's own hardware by default — nothing about a locally-run query has to leave the machine to get an answer.

It's worth being equally precise about what this doesn't solve on its own. AssistantGeneral also lets a user or an administrator's policy attach any of 9+ optional cloud model providers when that's the right tradeoff — for a task where cloud-model quality matters more than the data is sensitive, say. The moment that happens, the request is leaving the machine, and the exact same BAA and vendor-agreement questions from the first half of this post apply to that specific call, in full. On-device architecture removes the compliance question for the work that stays local by default; it doesn't retroactively make an opt-in cloud call exempt from the rules that call is actually subject to. The honest version of the pitch is "the default path avoids the question entirely," not "this product makes cloud AI compliant" — because nothing makes that automatic, on any product.

For anything at a real regulatory bar — clinical documentation with PHI, privileged legal work — the organization still needs its own governance layer on top: identity, audit, and policy over who can do what with which model, on-device or attached cloud one alike. That's a separate, real piece of infrastructure, not a side effect of running locally.