Distilled Models for Agentic Work: The Testing Methodology Procurement Cannot Skip

What the procurement question actually is Through Q1 and Q2 2026 the procurement conversation around distilled and open-weight models in regulated mid-market buyers has shifted from "can we use them at all", answered substantially in the DeepSeek compliance briefing and the distilled models procurement briefing, to "on which workloads are they actually fit, and how do we know." The first question is a contractual one and is now answered by the architecture decision (Architecture C self-hosted; hosted-API DeepSeek disqualified for HIPAA). The second question is operational and is the subject of this briefing. The procurement temptation is to treat the distilled-versus-frontier question as a benchmark comparison. The vendor publishes a comparison chart showing that the distilled model scores within a few percentage points of the frontier on a standard benchmark, MMLU, HumanEval, a domain-specific evaluation set, and the procurement team accepts the comparison as evidence the distilled model is fit for the buyer's workload. The acceptance is the structural failure. Standard benchmarks measure capability against a distribution of evaluation tasks that does not match the distribution of agentic work the buyer's workflow actually exhibits. A distilled model that passes the benchmark can fail the workload, and the failure pattern is consistent enough across our engagements to warrant a published methodology. The structural reason is one Anthropic and the broader research community have characterized in their disclosures about model distillation through 2025 and into 2026. Distillation works by training a smaller (or more efficient) model to imitate a frontier model's outputs across a large set of training examples. The resulting model is, on the central distribution of those training examples, very close to the frontier in capability. On the long tail, the workloads, contexts, and edge cases the training distribution underrepresents, the distilled model degrades unevenly and frequently sharply. Agentic workloads are exactly the long-tail case: they involve long horizons, tool calls whose outputs the model has to interpret, retrieval contexts whose provenance the model has to reason about, error states the model has to recover from. The agentic workload's failure-recovery distribution is what most distillation runs underrepresent because it is not what most evaluation benchmarks measure. For a regulated mid-market buyer the implication is that the testing methodology has to be the workload, not the benchmark. The distilled model is tested against the agentic workflow the buyer plans to deploy it in, against the buyer's data, against the failure cases the buyer has historical evidence for. The methodology is not exotic. It is also not what the vendor is going to run; the buyer (or the buyer's advisor) runs it. The three workload classes The methodology starts with classifying the candidate workload into one of three classes. The class determines the testing burden and the go/no-go thresholds. Class A, narrow structured high-volume. A workflow with a tightly constrained input format, a tightly constrained output format, a single-step or shallow multi-step model interaction, and high call volume against a stable distribution. Clinical documentation summarization following a fixed template, billing-code review against a structured input, transactional fraud-flag review against a structured set of inputs. Distilled models perform competitively on Class A workloads because the workload distribution is concentrated and the agentic surface is shallow. The testing burden is to confirm performance on the buyer's specific input distribution and to verify failure modes are detectable. Class B, moderate-horizon multi-tool routine. A workflow with a defined goal, two-to-five tool calls per execution, a moderately constrained set of valid execution paths, and a defined set of failure recovery patterns. Patient-intake intake summarization with EHR retrieval and structured output, contract-clause extraction with multi-document context, customer-support resolution with knowledge-base retrieval. Distilled models perform mixed on Class B workloads, they handle the central paths competently and degrade on the failure-recovery distribution. The testing burden is to map the failure distribution explicitly and to verify the model's behavior on each documented failure path. Class C, long-horizon multi-tool open-ended. A workflow with a goal that decomposes into a model-determined sequence of steps, an unbounded or large set of valid execution paths, tool calls whose outputs the model has to interpret in context, error states the model has to recognize and recover from, and a verification surface that itself depends on the model's reasoning. Multi-step clinical decision support with retrieval and reasoning across patient records, complex contract review with cross-document inference, multi-system reconciliation workflows. Distilled models reliably underperform on Class C workloads. The testing methodology for Class C is to confirm the underperformance pattern matches the buyer's tolerance and to make the architectural decision against frontier models early rather than late. The classification is the procurement screen's first artifact. A workflow's class determines whether the distilled-model conversation is even worth running. Class A is a productive conversation. Class B is a conditional one. Class C is, in most cases, the case for the frontier model with the existing BAA chain rather than the distilled alternative. The five evaluation dimensions For Class A and Class B workloads, the testing protocol evaluates the candidate distilled model along five dimensions against the buyer's actual workload. Each dimension produces a written score; the aggregate score determines the go/no-go. Dimension one, central-distribution accuracy. Run the candidate model against a representative sample of the buyer's actual workload, five hundred to two thousand examples drawn from production traffic with appropriate de-identification, and measure accuracy against the gold-standard outputs the buyer's existing process produces. The gold standard is the buyer's existing workflow's correct output, not the frontier model's output. The distilled model's job is to match the buyer's process, not the frontier model. Dimension two, long-tail accuracy. Run the candidate model against a curated set of edge cases the buyer has historical evidence for, atypical input distributions, low-frequency clinical presentations, unusual contract clauses, anomalous transaction patterns. The long-tail set is constructed from the buyer's historical exception logs, complaint logs, or escalation records. This is the dimension where distilled models most frequently underperform. Dimension three, tool-call fidelity (Class B). For workloads involving tool calls, evaluate whether the candidate model's tool calls are well-formed, parameterized correctly, and called in the right sequence. Measure tool-call invalid rates, parameter-error rates, and the frequency with which the model calls a tool whose output it then ignores or misinterprets. The benchmark is the buyer's workflow's existing tool-call patterns, not an idealized one. Dimension four, failure-recovery behavior (Class B). For workloads where tool calls or external systems can fail, evaluate the candidate model's behavior on injected failure cases, a tool returning an error, a retrieval returning an empty result, a downstream system returning a malformed response. The metric is whether the model recovers correctly, returns a structured stop reason, escalates appropriately, or produces a hallucinated output that papers over the failure. Distilled models frequently choose the last path; the testing surfaces it before production does. Dimension five, provenance and grounding (all classes). For workloads where the model's outputs have to ground in retrieved context or in user-provided input, evaluate the model's faithfulness, does the output reflect the retrieved context, does the output mark its sources, does the output distinguish between asserted-by-input and inferred-by-model. The metric is the rate at which the model produces outputs that the buyer's verification cannot trace to source material. Distilled models frequently produce ungrounded outputs at higher rates than frontier models because the training distribution underrepresents the discipline. The five dimensions, applied to a Class A workflow, produce a clear go/no-go. The five dimensions, applied to a Class B workflow, produce a conditional answer with a documented set of failure paths the architecture has to handle outside the model. The five dimensions, applied to a Class C workflow, produce the answer that the workflow should not be running on the distilled model in the first place. The architectural decisions that follow Three architectural decisions follow the testing methodology and frequently surprise procurement teams expecting a simple model-substitution answer. The distilled model frequently warrants a routing layer in front of it. The orchestration architecture covered in the governed-action shift briefing is the natural fit. The distilled model handles the central distribution of the workload at acceptable accuracy and substantial cost savings; the frontier model handles the long-tail cases the testing identified. The routing decision is policy-driven and recorded in the audit log. The result is a workflow that captures the distilled-model cost advantage on the bulk of traffic and the frontier-model capability advantage on the cases that warrant it. The verification harness becomes more important, not less. A distilled model whose long-tail failure rate is documented but non-trivial requires a verification surface that catches the failures before they propagate. The verification harness covered in the sabotage risk report briefing, structured stop reasons, denied-permission logging, provenance-aware context assembly, is the structural answer. A buyer who deploys the distilled model without the verification harness is paying for the cost savings with reduced confidence that the audit posture holds. The model lifecycle gets more active. Distilled and open-weight models update on the model authors' cadence and the buyer's pinned-version discipline has to be tighter than for the frontier-model BAA case. The buyer's posture is to pin the model version, run the testing protocol on every version upgrade, and document the version change as a §164.308(a)(8) change-management event. A quiet model update on a self-hosted distilled deployment is a §164.312(b) audit-controls finding waiting to be made. What the testing methodology surfaces that benchmarks do not We have run the methodology against distilled and open-weight model candidates on roughly a dozen mid-market healthcare and regulated-SaaS engagements through Q1 and Q2 2026. The findings cluster. The vendor's published benchmark comparison overstates the workload-relevant performance in the majority of engagements. The overstatement is not vendor mendacity; the benchmark distribution does not match the buyer's workload distribution, and the benchmark cannot tell the buyer that. The buyer's testing methodology is what closes the gap. The Class A workloads, narrow, structured, high-volume, frequently support the distilled-model substitution at substantial cost savings. We see fifty-to-eighty percent inference cost reduction on these workloads with no detectable accuracy degradation against the buyer's gold standard. The buyer's procurement file has a written justification for the substitution and a quarterly testing protocol to confirm the substitution remains fit. The Class B workloads frequently support a routed architecture, distilled model on the central distribution, frontier model on the long tail, with cost savings of twenty-to-fifty percent and an audit posture that holds. The architecture has more components than a single-model deployment; the cost savings justify the additional engineering and the verification harness becomes a load-bearing surface rather than an optional one. The Class C workloads almost never support distilled-model substitution. The testing methodology surfaces the underperformance early and the architectural decision moves toward frontier model with the established BAA chain rather than toward distilled with a constructed-on-the-fly governance surface. The buyer's procurement time is spent on the BAA-chain audit, not on the model substitution conversation that was never going to land. What we recommend A regulated mid-market buyer with an active distilled-model evaluation should run the testing methodology before the procurement conversation continues. First: classify the workload into Class A, B, or C. A workflow that the team cannot confidently classify is itself a workflow that warrants the conservative answer (frontier model on the existing BAA chain) until the classification is resolved. Second: for Class A and B candidates, run the five-dimension testing protocol against the buyer's actual workload data. The protocol is not exotic; the engineering function or the buyer's advisor can run it in a one-to-two-week scoped engagement. The artifact is a written go/no-go with the per-dimension scores. Third: for Class A go decisions, document the substitution in the procurement file with the testing protocol attached and a quarterly re-test commitment. The substitution is a real cost saving with a real audit posture; the documentation is the buyer's defense. Fourth: for Class B go decisions, design the routed architecture explicitly. The routing policy, the verification harness, and the long-tail escalation pathway are the components. The architecture has more pieces than a single-model deployment; the governed-action shift briefing and the agent control layer briefing cover the substrate. Fifth: for Class C decisions, return to the frontier-model BAA chain conversation and stop spending procurement time on the distilled-model substitution. The right answer is the existing BAA on the frontier model with the existing audit posture; the distilled-model conversation is a distraction the procurement team can set down. The distilled-model conversation is a productive one for the workloads where it is productive and a distraction for the workloads where it is not. The testing methodology distinguishes the two. The buyer who runs the testing before the procurement conversation has a written answer; the buyer who signs the contract on the vendor's benchmark comparison is the buyer who discovers the long-tail failure pattern in production.