HIPAA AI Architecture: A Reference Implementation for Mid-Market Healthcare

The architecture question is the question When a mid-market healthcare nonprofit, a multi-site behavioral health provider, or a regulated digital health SaaS asks us to help them adopt AI, the conversation almost always opens with the same question: "Is Claude HIPAA-compliant?" or "Can we use ChatGPT?" or "Is Bedrock the right bet?" The question is reasonable. It is also the wrong question. A model on its own does not have a HIPAA posture. A vendor's BAA covers what the vendor's BAA covers, usually a model endpoint, a data residency commitment, a logging behavior, and the obligation to notify in the event of a security incident. That is necessary. It is far from sufficient. The thing the Office of Civil Rights cares about, and the thing that gets cited in the audit findings we read every quarter, is the architecture the model sits inside: how prompts move from a clinician's interface to inference and back; what gets logged, where, for how long, with which access controls; what happens when the orchestration layer makes calls to other systems; and whether the integration with the EHR or the billing system retains its consent posture across the boundary. We have read more than fifty HIPAA assessment reports across mid-market healthcare in the last eighteen months. The pattern is consistent. The orgs that get cited are not the orgs that picked the wrong model. They are the orgs that picked a model with a clean BAA and then deployed it inside an architecture that the BAA did not actually cover. A BAA on the model endpoint does not retroactively cover a Slack channel where a nurse pastes a discharge summary into a prompt. A BAA on the model does not cover the audit log retention on the orchestration layer the engineering team assembled in two weeks. A BAA on the model does not cover the RAG pipeline that pulls from a vector database that nobody mapped against minimum-necessary. The architecture question is the question because the architecture is where the audit lives. The architecture is also where the decision compounds: an architecture chosen well in 2026 supports model swaps in 2027, regulatory change in 2028, and acquisition diligence in 2029. An architecture chosen badly is rebuilt twice in five years. This is a reference for getting the architecture right the first time. It walks through the seven HIPAA controls AI tools meaningfully touch, the four architecture shapes that actually exist for mid-market healthcare buyers, the four-vendor matrix as it stands at publication, the three failure modes we see most frequently in the field, and the 30-day audit method we run with clients on the Diagnostic. Where it gets technical, it gets technical only as far as a CIO needs to be able to lead the conversation with vendors and the auditor, not so far that it reads like a model card. The seven HIPAA controls AI tools must satisfy HIPAA's Security Rule lives in 45 CFR §164.308 (administrative safeguards), §164.310 (physical), and §164.312 (technical). The Privacy Rule and breach notification rules live in adjacent sections. Most of the rule does not change in an AI context, administrative safeguards are administrative safeguards regardless of what is doing the processing. But seven specific controls take on new meaning the moment an AI workflow touches PHI, and these are where the audit findings concentrate. We list them in audit-priority order. 1. Audit logging at the prompt level (§164.312(b)). The Security Rule requires an audit trail of access to PHI. In a traditional EHR, the audit trail is a database query log. In an AI workflow, the equivalent is the prompt-and-completion log: what was sent to the model, what came back, who initiated it, and at what time. Most AI deployments we audit log the API call but not the prompt content. That is an audit-trail gap. The control is not satisfied by "we have observability"; it is satisfied by "we can produce, on regulator demand, the exact text of every PHI-containing prompt sent to the model in the last six years." 2. Minimum necessary standard (§164.502(b)). The Privacy Rule requires PHI use to be limited to the minimum necessary for the purpose. In a RAG architecture, that means the retrieval step must filter context to the minimum required, not pull the entire patient chart because it was easier to embed. We have read assessments where the RAG pipeline injected three years of progress notes into a prompt asking the model to draft a single discharge summary. The model gave a fine answer. The architecture violated minimum-necessary on every call. 3. Access controls (§164.312(a)(1)). The model itself sees what the application sends it. The controls that matter are upstream: which clinicians can initiate which prompts; which prompts are gated by additional authorization; whether the orchestration layer enforces role-based access on the underlying data sources. "The model is fine because it has a BAA" misses the point. The model is downstream of the access decision. 4. Transmission security (§164.312(e)). PHI must be encrypted in transit. For AI workflows this is rarely the model API itself, every major provider serves over TLS, but it is frequently the integration layer between the orchestration service and the data source. We see plaintext SQL queries against the EHR's reporting database from orchestration services that the engineering team wrote on a Tuesday. Encryption in transit fails on the boundary, not on the model call. 5. Integrity controls (§164.312(c)(1)). PHI must not be altered or destroyed in unauthorized ways. AI introduces a specific risk here: model output that updates a record. If a clinical-document generation workflow writes a draft note back to the EHR, the integrity control must distinguish a clinician-authored note from a model-authored draft. The audit looks for this distinction. Most deployments don't have it. 6. Person or entity authentication (§164.312(d)). The clinician's identity must be tied to the prompt, the completion, and any downstream action the AI takes on their behalf. SSO from the EHR to the AI tool is necessary. SSO that loses the user identity at the orchestration layer (because the orchestration layer makes its own service-account calls) is the most common implementation mistake we see. The audit log shows "ai-service-account" instead of "Dr. Singh." 7. Business associate management (§164.308(b)). Every entity in the chain that touches PHI must be under BAA. The model vendor's BAA is one link. The orchestration vendor (LangChain Cloud, Vellum, Vercel AI, etc.) is a separate link. The vector database vendor (Pinecone, Weaviate Cloud) is a separate link. The observability vendor (Datadog, Honeycomb) is a separate link if it ingests prompt content. The control is the chain, not the model. We have read twelve-page architecture reviews where every component had a BAA-eligible posture and three of them did not have a BAA actually executed. That gap surfaces in the next audit. The seven controls above are not exhaustive, HIPAA has more to say about disaster recovery, contingency planning, workforce training, and physical safeguards, but they are the seven that the architecture decision either satisfies or quietly violates. Every control on this list is a question a vendor's BAA does not, on its own, answer. Every control is a question the architecture either gets right or gets wrong. The four-architecture decision tree There are exactly four architecture shapes a mid-market healthcare organization can choose for AI. We have not seen a fifth that holds up. The choice is not driven by the model. It is driven by where PHI lives, where it is allowed to go, and what the regulator-facing log looks like. Architecture A, Hosted SaaS with full BAA. The clinician uses a vendor application (AI scribe, ambient documentation, AI medical coding assistant) where the vendor takes the model call, the orchestration, the audit log, and the data residency on their side, all under a comprehensive BAA. PHI moves from the clinician interface to the vendor's environment and back. The healthcare org's responsibility is procurement-level: confirm BAA scope, confirm SOC 2 Type II, confirm breach notification SLA, confirm data residency, confirm minimum-necessary configuration. When this works: narrow, well-defined clinical workflows where the vendor has already encoded the audit trail (Abridge, DeepScribe, Suki, Nuance DAX). When this does not work: anything outside the vendor's scoped product, any custom workflow, any ROI-driven internal tool. Buying SaaS for everything ends in 14 vendors and a BAA-management problem. Architecture B, Cloud-native with self-managed BAA scope. The healthcare org builds its own AI workflow on a major cloud, typically AWS Bedrock or Azure AI Foundry, where the cloud provider's BAA covers the full set of services in scope (model, orchestration, storage, vector DB, observability). PHI moves only between cloud-provider services that are explicitly enumerated in the BAA addendum. The healthcare org's responsibility is engineering: the architecture documents must show every service in scope, the BAA addendum must enumerate those services, and the audit log must demonstrate that PHI never left the BAA-covered service set. When this works: healthcare orgs with engineering capacity (10+ engineers), a multi-year AI roadmap, and the discipline to keep PHI on-cloud. When this does not work: orgs that "just want to try Bedrock real quick" without committing to the engineering rigor, which is most of them. Architecture C, On-premises or VPC-isolated. The model runs in the healthcare org's own infrastructure (on-prem, customer VPC, single-tenant cloud) and PHI never leaves the trust boundary. Open-source or weight-licensed models (Llama, Mistral) on dedicated GPUs, plus orchestration in the same network segment. When this works: large, well-resourced healthcare systems with existing infrastructure teams and explicit regulatory pressure to keep PHI on-prem (some state Medicaid programs, some VA workflows, some 42 CFR Part 2 substance-use programs). When this does not work: mid-market orgs that don't have the operational maturity to run model inference at the SLA their clinicians expect. The economics rarely pencil out below 1,000 employees. Architecture D, Hybrid: PHI never leaves the trust boundary; AI never touches PHI directly. A non-PHI representation (de-identified summary, structured codes, abstraction layer) is what crosses to the model. The model returns a non-PHI response, which is then re-identified or applied inside the trust boundary. When this works: policy generation, regulatory analysis, audit-evidence drafting, board-document prep, workflows where the model's job is content production from non-PHI inputs. When this does not work: workflows where the model needs the patient-specific clinical context to be useful (clinical decision support, care-pathway generation). This architecture is underused; it is also the safest place to start for an org that has not yet committed to A, B, or C. The decision tree is straightforward. If the use case is narrow and a vendor sells it: A. If the org has the engineering rigor and a roadmap: B. If the workload genuinely cannot leave the trust boundary: C. If the org wants to ship something useful in 90 days without committing to A, B, or C: D, and use D as a stepping stone to whichever of A, B, or C is the long-term shape. Vendor matrix: Anthropic, OpenAI, AWS Bedrock, Azure AI Foundry The four-architecture decision frames the vendor question. Within Architecture B (cloud-native with self-managed BAA scope), the active vendor decision in 2026 is between four model providers. We score each on the dimensions that determine HIPAA-defensibility, not the dimensions that determine model performance. Performance is table stakes; HIPAA posture is what separates them. | Dimension | Anthropic (direct) | OpenAI (direct) | AWS Bedrock | Azure AI Foundry | |---|---|---|---|---| | BAA available | Yes (enterprise, on request) | Yes (enterprise, with Zero Data Retention addendum) | Yes (under AWS BAA) | Yes (under Microsoft BAA) | | Data residency commitment | US (with regional commitments on enterprise) | US (with explicit Zero Data Retention required) | Per-region, enumerated in BAA | Per-region, enumerated in BAA | | Training-data exclusion (no model training on inputs) | Default for API; explicit in BAA | Requires Zero Data Retention; default for API | Default; explicit in BAA addendum | Default; explicit in BAA addendum | | Audit log access (prompt-level) | Customer-managed (you log) | Customer-managed (you log) | CloudTrail + Bedrock invocation logs | Foundry traces + Azure Monitor | | Supported model versions covered by BAA | Claude family (current and prior generations) | GPT-4o, GPT-4 Turbo, GPT-4, embedding models | Claude, Llama, Mistral, Amazon Nova, Cohere, Stability | Azure OpenAI (GPT-4o, GPT-4), open-weight models in Foundry catalog | | Orchestration / agent layer covered | No (you build it) | Limited (Assistants API; check BAA scope) | Bedrock Agents, Bedrock Knowledge Bases (under BAA) | Foundry Agents, AI Studio (under BAA) | | Vector DB covered | No (separate vendor) | No (separate vendor) | OpenSearch + Bedrock KB (under BAA) | Azure AI Search (under BAA) | | Best fit architecture | B (with strong engineering team) or D | B or D | B (most coverage out of the box) | B (most coverage if you're already on Azure) | Two practical observations from the matrix. First: Bedrock and Foundry have the broadest BAA scope because they cover the orchestration and vector layers natively. For Architecture B, that meaningfully reduces the number of separate BAAs the healthcare org has to manage. Anthropic and OpenAI direct API both require the customer to bring their own orchestration and vector DB, with separate BAAs for each. That is workable; it is also the source of most of the BAA-chain gaps we audit. Second: training-data exclusion on OpenAI direct API requires the explicit Zero Data Retention addendum. Without it, OpenAI's default API terms allow them to retain inputs for abuse monitoring (with 30-day default retention). For PHI workloads, ZDR is non-negotiable, and the BAA conversation must include it. We have seen healthcare orgs assume their enterprise OpenAI agreement included ZDR when it did not. The matrix changes quarterly as the vendors update their offerings. The dimensions do not. We update our internal version of this matrix every 90 days and republish; the latest is in the BAA Coverage Matrix download paired with this guide. The three failure modes Securem sees in the field We have run the HIPAA AI architecture review on more than thirty engagements. Three failure modes recur. Each is sanitized below from real findings. Failure mode 1: BAA covers the model, not the orchestration. A regional behavioral health provider deployed an internal AI tool to generate progress notes from session transcripts. The model was Claude via Anthropic's direct API, BAA executed. The orchestration was a Vercel-hosted Node.js service that accepted transcripts, called the model, and returned drafts. Vercel had no BAA. PHI passed through Vercel's infrastructure on every request. The provider had answered "yes, we have a BAA" on the procurement checklist truthfully. The audit finding was a six-figure remediation. The fix was migrating the orchestration to AWS (already had a BAA) and rewriting the deployment in two weeks. The lesson: every component of the request path must be on the BAA chain, not just the named model vendor. Failure mode 2: Audit log captures the API call, not the prompt content. A healthcare nonprofit deployed an AI summarization tool over their patient communication portal. Datadog observability captured the API request count, latency, error rate, and model version. It did not capture the prompt body or the completion. When the OCR investigation arrived (unrelated to AI; a separate breach), the regulator asked for the prompts that had referenced a specific patient over the prior twelve months. The org could produce request counts. They could not produce content. The deficiency added a finding to the otherwise clean response. The fix took six months, a custom logging layer on the orchestration service that captured prompts and completions to a long-retention immutable store, separately from operational metrics. The lesson: prompt content is the audit log, not metadata. Failure mode 3: RAG context window leaks PHI into adjacent non-PHI logs. A regulated SaaS serving healthcare customers built a customer-support AI assistant. The retrieval layer pulled context from the support knowledge base (non-PHI) and from the customer's recent ticket history (which sometimes contained PHI when customers had escalated patient-facing issues). The model's full prompt, including the retrieval context, was logged for prompt-engineering iteration. The prompt log was an S3 bucket without HIPAA-grade access controls (the team building the AI assistant didn't think they were handling PHI). PHI ended up in a non-BAA-covered log. The remediation was scrubbing five months of prompt-engineering logs and rebuilding the logging pipeline to redact PHI before storage. The lesson: any path PHI can travel must be assumed to carry PHI until proven otherwise, and the logging and storage on that path must match. The pattern across all three: the model was not the problem. The architecture around the model was the problem. In each case, the BAA on the model endpoint was real and complete. In each case, the audit finding was elsewhere, the orchestration, the logging, the retrieval. In each case, the engineering team that built the AI workflow did not have HIPAA experience and the compliance team that had HIPAA experience was not in the architecture review. The 30-day HIPAA AI Architecture Audit When a healthcare organization engages us on the Diagnostic for Adopt-AI-Safely, we run a 30-day audit against the architecture they have or are about to commit to. The audit produces a written report that the CIO can hand to the board, the auditor, or the next vendor. The structure of the audit is the structure below; we publish it here because it is more useful as a reference than as a sales pitch, and because most of it can be done internally if the org has the bandwidth. Days 1–5: Scope and inventory. We document every AI workflow in production or planned for the next twelve months. For each: the use case, the data classification, the architecture shape (A/B/C/D), the model, the orchestration, the vector layer, the logging path, and the BAA chain. Output: a single architecture diagram per workflow, plus a master inventory. Days 6–10: BAA chain audit. We pull every BAA for every component on every workflow's request path. We check for: scope (which services are covered), data residency (which regions), training-data exclusion (ZDR or equivalent), breach notification SLA, audit-rights, sub-BAA flow-down for any sub-processors. Output: a BAA gap register, every component on the chain that does not have a BAA, an incomplete BAA, or an out-of-date BAA. Days 11–15: Control-by-control review. We walk the seven controls in section 2 against each workflow. Audit logging at the prompt level, present or absent, retention duration, access controls. Minimum necessary, what does the retrieval layer actually inject. Access controls, does the orchestration preserve clinician identity. Transmission security, encrypted on every hop, including internal hops. Integrity, clinician-authored vs. model-authored. Person/entity authentication, does the audit log show the clinician's identity, not a service account. Business associate management, gap register from days 6–10. Output: a control matrix, workflow × control, with status (Met / Partial / Missing). Days 16–20: Failure-mode walk-through. We run each workflow against the three failure modes plus seven secondary patterns we track internally. For each failure mode, the question is "could this happen here, and how would we know if it did." Output: a failure-mode inventory with detected exposures. Days 21–25: Remediation prioritization. We rank every gap from days 11–20 by audit-finding likelihood × cost-to-remediate. We give the org a sequenced fix list with owner, effort estimate, and timeline. The list typically has 5–15 items; for orgs that built carelessly, it has more. Output: prioritized remediation plan. Days 26–30: Report and walkthrough. We deliver a 25–40 page written report. The CIO can hand it to the board. The auditor can read it. The next vendor sees what the audit covered. We run a 60-minute walkthrough with the leadership team and provide a 30-day clarifying-question window. The report stands on its own, it does not require us to act on it. The 30-day audit is what the Adopt-AI-Safely Diagnostic is. It is a fixed-scope, fixed-price written assessment. It is not implementation. It is not a retainer. It is the audit-grade architecture review that tells you whether your AI program is HIPAA-defensible, and if it is not, what specifically to fix and in what order. What to do this quarter Three actions any healthcare CIO can take this month, regardless of whether they engage us: 1. Inventory every AI workflow and proposed workflow. Pull the list from procurement (vendor SaaS), engineering (internal builds), and the practice leaders (shadow IT). Map each to one of the four architecture shapes (A, B, C, D). Most orgs we work with cannot do this exercise in a single afternoon, that's the first finding. 2. Audit the BAA chain for the highest-exposure workflow. Pick the workflow that touches the most PHI. Pull every BAA for every vendor on the request path. Confirm scope, residency, ZDR. Most orgs find at least one gap on the first workflow they audit. 3. Add a prompt-content log to the highest-exposure workflow. If the org's AI tools log API calls but not prompt content, fix that one workflow first. Long-retention, immutable, access-controlled. The Security Rule audit log requirement is non-negotiable; the implementation on AI workflows is the gap. Three actions, ninety days, no engagement required. If the inventory or the BAA chain or the logging surface gaps the org can't close internally, or if the architecture decision is large enough that getting it wrong is expensive, that's where the Diagnostic comes in. Two to three weeks, ten thousand dollars, written report you keep regardless. The BAA Coverage Matrix download paired with this guide gives you a side-by-side scorecard for the four major AI providers across the seven dimensions in section 4, plus a worksheet for adding your own vendor BAA data. Use it on every AI procurement conversation. Use it on every architecture review. Update it quarterly as the vendor landscape moves, and we'll keep doing the same on our side.