Repatriating Regulated Inference: When Open Weights Make the Perimeter the Right Place for the Model

If you run technology for a behavioral health group, a healthcare nonprofit, a PE portfolio company, or a regulated SaaS business, you have spent the last two years answering one question the same way: how do we use a capable model without letting regulated data leave our control? The default answer was a managed API and a signed BAA, and for most workloads that is still correct. But there is a class of data where trusting the vendor's boundary was always a compromise you tolerated rather than chose, the PHI a HIPAA obligation actually attaches to, the specific data categories you named in your SOC 2 commitments, the financial-services data behind Treasury's controls. For that class, the building blocks to keep the model inside your own boundary just arrived in usable form, and the decision is now yours to make on the merits instead of deferring it as exotic. The on-device and in-perimeter basics, why a private inference path matters, what a no-egress boundary buys you, we covered before, and I won't restate them. What changed is that the build-versus-buy line moved, and the new decision a regulated buyer owns is not whether to self-host. It is data-class routing: deciding, per category of data, where inference is allowed to happen. What changed: open weights met cheaper accelerators Two things moved at once. On May 1, Google released the Gemma 4 open-weight models, weights you can download, run, and keep entirely inside infrastructure you control. Open weights are not a new concept, but the capability gap that used to make self-hosting a quality sacrifice has narrowed to the point where, for bounded enterprise tasks like summarization, extraction, classification, and retrieval-grounded answering over your own records, a self-hosted open-weight model is a serious option rather than a downgrade you apologize for. At the same time, the cost of the accelerators you'd run those weights on has been trending down. I am deliberately not going to quote you a GPU rental price or a payback-in-N-months figure, because those numbers are wrong the week after they're published and because your real payback depends on your token volume, your existing cloud commitments, and your staffing. The qualitative shift is what matters: open weights plus falling accelerator costs mean in-perimeter inference is increasingly viable for steady, predictable workloads, where a year ago it penciled out only for the largest shops. Put those together and the perimeter becomes a legitimate place to put the model, not for everything, but for the data that was never supposed to leave in the first place. Why the old control model misses it The control model most regulated teams built treats the AI vendor as a third party you manage with paper: a BAA, a sub-processor list, a vendor risk review, a data-processing addendum. That model is fine when the alternative is no control at all. Its blind spot is that it assumes the only two options are send the data to the vendor or don't use AI. It never priced in a third option where the model runs inside your boundary and the third-party-transfer question simply does not arise. The managed-API world has gotten better at attribution and isolation, Bedrock added request-level usage attribution on May 20, and Microsoft Foundry shipped project-level cost attribution with Managed VNet GA on May 31, and those are real improvements for governance and chargeback. But attribution tells you who spent what; it does not change the legal fact that the data crossed a boundary into someone else's environment. There is also a quieter risk in the pure-managed posture: model pinning. When GPT-5.5 Instant became the ChatGPT default on May 5, it was exposed through a floating chat-latest alias, the model behind that name can change with no notice. With frontier labs shipping on a roughly six-week cadence (Opus 4.8 landed about 41 days after 4.7), a regulated workload pinned to a moving alias is validating against a target that shifts under it. For your perimeter-bound classes, the open-weight path gives you a version you froze and can reproduce. What the audit and the board will ask Once you can keep some inference in-perimeter, your auditor and your board stop accepting we use a compliant vendor as the whole answer. Expect these questions, and have the artifact ready before they're asked. Where does each data class actually go? You will need a routing map: which categories are perimeter-bound and which are cleared for a managed API, with the rule written down rather than living in an engineer's head. Treasury's Financial Services AI RMF, 230 control objectives across seven domains, published February 19, is explicitly the kind of framework that expects you to know, per data category, where processing happens. What enforces the boundary? Intending not to send PHI out is not a control. The board will ask what technically prevents egress from the in-perimeter path, and who could change it. Whose account holds the weights and the keys? For the perimeter-bound path, the answer they want is: your account, your VPC, your KMS keys. If your model changes, how do you know? Version pinning, change logs, and re-validation on update, the discipline you could not impose on a floating managed alias. The artifact: a data-class routing decision and an in-perimeter path The concrete thing to build is a one-page data-class routing decision plus the architecture that enforces it. The routing decision lists your data categories and assigns each a destination. Perimeter-bound classes, HIPAA PHI, the specific categories named in your SOC 2 commitments, the data behind your Treasury controls, route to a self-hosted open-weight model. Everything else, public content, internal-but-not-regulated text, low-sensitivity operational data, routes to a managed API, where you get frontier capability and someone else's reliability budget. The point is not to repatriate everything. It is to repatriate the narrow set where transfer is the actual risk, and keep the managed path for the wide set where it isn't. The in-perimeter architecture for that narrow set has four properties worth stating plainly: the open-weight model runs in a private VPC with no egress to the public internet; the weights live in your own cloud account, not a vendor's; access goes through your identity and key management, so the same audit trail that covers your databases covers the model; and the version is pinned and logged, so an auditor can see exactly which weights answered which request. If you want a worked reference for how the surrounding controls fit together, our HIPAA AI Architecture reference implementation walks the full pattern: /blog/hipaa-ai-architecture-reference-implementation. What we recommend Write the routing map first. Before any infrastructure decision, list your data classes and assign each to perimeter-bound or managed-API. Most teams discover the perimeter-bound set is smaller than they feared, which makes the project tractable. Run the payback math for your own environment. Take your real steady-state token volume for the perimeter-bound classes, your existing cloud commitments, and your staffing cost, and model in-perimeter versus managed for that slice only. Do not borrow anyone else's numbers, including a vendor's. The answer is environment-specific by design. Make egress prevention a tested control, not an intention. Whatever you stand up, verify that the in-perimeter path cannot reach the public internet, and put that test in your evidence pipeline. Pin and reproduce. Freeze the open-weight version for regulated workloads, log it, and re-validate on every change, the discipline the managed alias world cannot give you. If you want a second set of eyes on the routing decision before you commit capital, that boundary-and-routing review is exactly the shape of our fixed-scope Diagnostic. Decide where each data class runs before your default vendor decides for you.