On-Device AI for the Regulated Professional: The Two-to-Three-Year Window the Vendor Stack Has Not Filled

What changed in the cost structure Through 2024 and 2025 the dominant assumption in regulated AI procurement was that frontier inference happened in the cloud, on a hyperscaler's tenant, under a BAA the buyer extended through one of the four architectures named in the HIPAA AI Architecture Field Guide. The exception, Architecture C, on-premises or VPC-isolated, was treated as a heavyweight option for the small minority of workloads whose data classification absolutely required it: classified-adjacent federal contractor work, certain attorney-client matters, the most sensitive psychiatric and substance-use records under §42 CFR Part 2. Two things changed in late 2025 and the first months of 2026 that move Architecture C from the heavyweight exception to a viable mainstream option for a defined band of regulated workloads. The first is the cost structure of cloud inference. Frontier-model inference at enterprise volume is, in 2026, a variable-cost line item that scales with usage and that has shown sharp upward repricing on a six-month cycle. The math that worked when a workflow ran a few thousand calls per month against a gpt-class model, the math the procurement file budgeted against, does not hold at the volume the same workflow exhibits in steady-state operation, where inference costs frequently outrun the seat license that procured the workflow in the first place. The shift is the cloud-software analog of a meter the buyer did not negotiate with the procurement file open. The second is on-device capability. Apple's silicon roadmap, the open-weight model class led by Llama and the smaller Qwen and Mistral models, and the maturation of llama.cpp / mlx / vLLM as the local-inference runtimes have, by mid-2026, produced a capability profile where a Mac Studio or an M-series Mac Mini cluster can run models in the seven-billion to seventy-billion parameter range at quality acceptable for a wide band of professional workflows, clinical documentation summarization, legal contract review, accounting workpaper review, fiduciary research summarization. The capability is not at frontier-model parity for arbitrary reasoning; it is at frontier-model parity for the structured, repeatable, narrow-domain workloads that constitute most of the work in those professions. The capability is also legally simpler, the inference happens on hardware the buyer owns, in a building the buyer controls, against data that never leaves the trust boundary. There is no BAA chain to construct because the cloud vendor is not on the path. Where the regulated mid-market buyer is today What we see across our Q1 and Q2 2026 engagements with mid-market healthcare, regional law firms, regional accounting practices, and registered investment advisers is that the buyers in this band are not waiting for vendor infrastructure to arrive. They are improvising. The most common pattern is a small Mac cluster, frequently a Mac Studio paired with two Mac Minis, or three M-series Mac Studios, running a local inference runtime against an open-weight model, fronted by a custom API the buyer's engineering function or contracted developer built. The workflows the cluster handles are the ones the buyer's general counsel, compliance officer, or §164.308 risk analysis flagged as ineligible for cloud-routed inference under the data classification framework. The improvisation works as a stopgap. It does not work as a posture. The hardware is consumer-grade, not rack-mounted; the clustering software is a community project; the operational discipline (patching, monitoring, capacity planning, backup, change management) is whatever the buyer's IT function can spare from its primary workload. There is no vendor on the relationship that can be held to a service level. There is no BAA, but more importantly there is no vendor BAA that needs to exist because the cloud vendor is not on the path, yet there are still real audit-controls obligations under §164.312(b) that the buyer's IT function is now solely responsible for satisfying. The product gap is direct: rackable Apple silicon, with clustering software designed for the workload, with a hardware-vendor relationship that includes service levels, with an integrator ecosystem that can stand the workflow up to enterprise standards. Apple has not shipped that product. No third party has shipped it at scale either, though several startups are in the early stages. The two-to-three-year window before the gap is filled is real. What the procurement question becomes For a regulated mid-market buyer in healthcare, law, accounting, or wealth management with an active AI procurement decision in 2026, the question is no longer which cloud vendor's BAA do we extend. It is which workloads can route to cloud, and which workloads have to stay on-device, and what is the architecture and the operational discipline for each. The triage we run on the Adopt-AI-Safely Diagnostic uses three criteria to sort a workload into Architecture A through D from the HIPAA AI Architecture Field Guide, with on-device inference being the default for workloads that surface high in any of the three. Criterion one, data classification beyond the standard PHI floor. Workloads touching §42 CFR Part 2 substance-use records, attorney-client privileged material, work product subject to the trial-prep exception, federal contractor restricted data, or fiduciary-confidential research warrant a higher data-residency posture than the standard HIPAA BAA contemplates. Architecture C (on-prem) and Architecture D (hybrid with PHI never leaving the trust boundary) are the natural fits. Criterion two, concentration of high-volume routine inference. A workflow whose steady-state inference volume runs in the high tens of thousands of calls per month at a per-call cost the buyer's budget cannot absorb is a workflow that pays for its own on-device hardware in months. The math has shifted from "on-device costs more" to "on-device pays back inside the procurement cycle." Clinical documentation, billing-coding review, contract review, and standard transactional summarization are the workloads where this math frequently surfaces. Criterion three, vendor-lock-in concern as a strategic input. A buyer who is concerned about middleware-vendor exposure (covered in the middleware trap briefing) and frontier-model substitution risk frequently prefers an architecture where the model is a configurable, swappable component rather than a contract commitment. On-device inference against open-weight models is the cleanest articulation of that posture: the model can be swapped in a day; no BAA chain breaks; the workflow does not change. A workload that surfaces high on any one of the three criteria is a candidate for on-device inference even where the standard BAA-extended cloud route would technically work. A workload that surfaces high on two or three is one where the on-device route is the structural default and the cloud route is the conditional exception. What the operational posture looks like The buyers we work with who have crossed into Architecture C with a Mac-cluster substrate face a defined set of operational decisions that the cloud-routed posture did not surface. The decisions are answerable, but they have to be answered. Hardware lifecycle. Apple silicon Macs do not ship with the warranty terms, replacement program, or hot-swap discipline that a server-class buyer expects. The cluster needs documented lifecycle management: a primary, a hot spare, a documented replacement procedure, and a quarterly drill against the procedure. The buyer's IT function or its managed-service provider owns this; no vendor will own it for the buyer in the next 12 months. Model lifecycle. Open-weight models update on the model authors' cadence, not the buyer's. The buyer's posture is to pin a model version against the workflow, run a versioned upgrade discipline that includes regression testing against the buyer's own test set, and document the model change as a change-management event under §164.308(a)(8). The model on the Mac cluster today is the auditor's "what is the system" answer; a quiet model swap is a §164.308(a)(8) finding waiting to be made. Audit-controls discipline. The §164.312(b) requirement that the system maintain an audit trail does not relax because the system is on-device. The audit log has to capture the same fields the cloud-routed audit captures: prompt, response, user identity, timestamp, model version, and the structured stop reason described in the code execution audit trail briefing. The audit log lives on hardware the buyer controls, frequently a separate, hardened logging surface, and is retained on the same six-year retention window as other audit-controls evidence. Capacity and queueing. A Mac cluster's throughput is finite; the workflow has to handle the queue gracefully. A clinical documentation workflow that times out on a Friday afternoon because the cluster is at capacity is a §164.308(a)(7) availability finding. The buyer's posture is to monitor the queue, alert on saturation, and have a documented overflow path, frequently a manual-completion fallback rather than a cloud-routed overflow that would defeat the data-residency posture. Network containment. The Mac cluster sits inside the buyer's trust boundary, not behind a hyperscaler's network. The network controls, segmentation, ingress/egress policy, DNS filtering, monitoring, are the buyer's responsibility and have to be designed against the same threat model the buyer would extend to any production system handling regulated data. The operational tax is real. It is also bounded, the disciplines above are familiar to any IT function that runs an EHR, a practice-management system, or a regulated-data warehouse. The tax is the price of the architecture; the architecture is the legally simpler position for the data classification. What we recommend A regulated mid-market healthcare, law, accounting, or wealth-management buyer with an active AI procurement decision in the next two quarters should treat on-device inference as a first-class option for the workloads where the data classification or the cost math justifies it. First: triage the workload portfolio against the three-criterion screen above. Most buyers find that 60-to-80% of workloads route to cloud Architecture A or B without controversy and that 20-to-40% surface a criterion that warrants Architecture C or D consideration. Second: for the on-device candidates, scope the operational tax honestly. Hardware lifecycle, model lifecycle, audit-controls discipline, capacity management, and network containment are the five dimensions. The buyer's IT function has to staff against them or contract for them; pretending the operational cost is zero is the path that produces the §164.312(b) finding two years out. Third: do not paper over the architecture gap with a hosted-API BAA the data classification cannot legally support. We see this pattern repeatedly: a workload that should be Architecture C gets routed to a hyperscaler under a BAA, the procurement file documents the BAA, and the data-classification finding sits unresolved waiting for an audit. The hosted BAA does not cure the underlying classification mismatch; it shifts the finding from "the data is in the wrong place" to "the data is in the wrong place under contract." Fourth: track the vendor stack. Apple's enterprise positioning, the rackable-silicon startups, and the clustering-software ecosystem are moving. The buyer who has the workflow architected around on-device inference today is the buyer who can adopt the vendor-shipped infrastructure when it lands, with no architectural rework. The buyer who routed to cloud "for now" is the buyer who has to re-architect when the regulator's question reaches the data classification. The two-to-three-year window before vendor infrastructure fills the on-device gap is real. The mid-market buyers who use the window to architect for the structurally simpler position are the buyers whose audit posture will hold through the next regulatory cycle. The Adopt-AI-Safely Diagnostic includes the on-device triage as a standing component of any engagement involving workloads where the data classification raises the question. The architecture is mature enough to commit to. The operational discipline is familiar. The procurement file's primary risk is no longer the cloud vendor's BAA; it is the data-classification mismatch that the BAA cannot cure.