Engineering Teams in the Age of Agentic AI
For years, the default approach to building ML products that solved business problems was to establish a research-heavy organization, with…
Engineering Teams in the Age of Agentic AI
For years, the default approach to building ML products that solved business problems was to establish a research-heavy organization, with data scientists and PhDs, sometimes working separately from the team that shipped the core product. These teams would consist of dozens of individuals. I lived that model from 2014 to 2019, running the core platform team, then a machine-learning / analytics team focused on network anomaly detection. It worked; up to a point. We built models and solid metrics, but translating research into durable, low-friction products was slow and expensive. Explainability was a challenge and still is.
The landscape shifted with LLMs. Over the last couple of years, I’ve built LLM-based applications (Bedrock and a mix of frontier/smaller models) with a much smaller team, just a handful of engineers and a single embedded subject-matter expert (SME) from the workflow we’re automating. We also moved from traditional ML metrics to a hybrid approach that includes a second model evaluating the first for truthfulness, faithfulness, and task success. The result: faster iteration, clearer ownership, and higher-quality outcomes that have built-in explainability from day one. Here’s the operating model that works and why it’s the future, for now.
Think of Agents as Software With More Autonomy
There’s a temptation to turn “agentic” into a research project. Resist it. Keep the philosophy simple and secure. Treat it like any other 3rd party or dependency you may be integrating into a system. Focus on shipping first, then iterate to improve against your metrics, which should guide you towards a better customer experience. Limit the action space by having agents use typed tools with clear pre- and post-conditions. These tools operate with minimal privileges, scoped credentials, and egress allowlists, avoiding open internet access by default. A deterministic framework guides the process, while a probabilistic core manages ambiguity, with state machines or graphs orchestrating the flow and LLMs handling uncertainty within set steps. Reversibility is standard: any destructive actions go through review gates until they meet the requirements for confidence and policy compliance. Secure tool design involves input validation, output sanitization, sandboxed execution, and restrictions on dangerous primitives like arbitrary shell commands or raw SQL unless tightly controlled. If you can’t sketch the agent’s states on a whiteboard in five minutes, it’s too complex. If you can’t explain its permissions in two minutes, it’s too risky.
A Hybrid Scoreboard That Actually Moves the Business
Classic ML metrics still matter. Starting with labeled data can be really useful, but it’s incomplete for LLM systems. The scoreboard spans three layers: Business outcomes include task success rate, cycle time saved, cost per unit, deflection rate, and SLA hit rate. Model behavior focuses on truthfulness/faithfulness, safety, adherence to instruction, and uncertainty calibration. System and security health cover latency budgets, tool-call error rate, fallback/handoff rate, PII/secret leakage rate, prompt-injection block rate, jailbreak attempt rate, retrieval poisoning detection, cross-tenant access violations, and audit completeness. This scoreboard is always on, so you need to set up a system that evaluates responses in near real time to ensure your solution remains true to the guardrails you have added.
The Evaluator Model Keeps the Primary Honest (and Safe)
Our most valuable enhancement was establishing a dual-model loop: The main model generates an answer or action, while the evaluator model verifies claims against evidence, assesses faithfulness and policy compliance, and advises on the next steps — whether to approve, request additional retrieval, escalate to a human, or revert to a safer route. Think of the evaluator as a typed brake pedal. It doesn’t make your system brilliant, but it ensures trustworthiness by catching hallucinations, prompt injections, privacy breaches, and policy violations at machine speed.
Shipping Discipline: It’s still just software
Small teams win by shipping, period. Rolling out applications that use LLMs should be no different in principle than any other application, with a few additional considerations. The habits below compound: Shadow → canary releases before a broad deployment. Prompts, evaluators, policies, and guardrails are maintained as versioned artifacts. Data contracts specify retrieval source parameters such as freshness, access control, and structured fields that evaluators can verify — ensure your model does not return unstructured text. Prioritize observability: traces connect inputs, tool calls, outputs, evaluator decisions, and costs into a single, audit-ready timeline. Monitor input and output tokens over time to detect approaching hard limits or operational changes — remember, input tokens often correlate with latency and costs, so stay vigilant. Incorporate security pre-mortems and red teaming into your sprint, attacking your system with injections, jailbreaks, and poisoned documents before customers do.
Deployments: Tier-1 Discipline for LLM Systems
My deployment view was forged in Tier-1 billing systems where zero deviation was the rule. Every upgrade ran in parallel on real production traffic, and we didn’t flip until the new system matched the old on the surfaces that mattered. LLM systems deserve the same rigor, especially because trust in their outputs is still fragile.
The philosophy:
Decide what must never change. Decide what can not change: (prices, invoice line items, policy decisions, approvals, privacy boundaries, tenant isolation, audit completeness). Those require parity before rollout. Creativity lives elsewhere.
Prove it in parallel by mirroring live requests to the candidate pipeline in read-only shadow mode. Compare outcomes continuously on critical surfaces, not just in a prelaunch batch.
Use semantic parity where words differ. Exact matches are easy; the hard part is “same meaning, different phrasing.” Evaluator-driven checks judge faithfulness, policy compliance, and security invariants.
Promote autonomy slowly. Start with read-only recommendations, then proceed to write behind (where the incumbent still commits), and finally move to write through when the behavior is reliably consistent.
Guardrails are operational, not aspirational. Circuit breakers roll back immediately if parity slips or security risk rises. Rollbacks are measured in minutes.
_Make the change explainabl_e. Lock versions of prompts, models, indices, tools, and evaluator policies. Each release leaves an artifact trail, so any output is reproducible and auditable.
Measure what customers feel. Parity is necessary but not sufficient; it’s important to monitor cycle time, prevent errors, and ensure customer satisfaction. If those don’t move, the change doesn’t matter.
In short: parallel-run until boring, then promote; and treat security invariants as part of the zero deviation contract.
Org Design: Fewer Meetings, More Ownership
The software development manager oversees the program’s operations, including weekly releases, clear exit criteria, practiced rollbacks, and security sign-offs integrated into the checklist. The SME acts as the product owner for accuracy and usefulness, providing real evaluative edge cases, including “don’t do this” scenarios. Engineers are responsible for the evaluation stack and guardrails just as much as for the app. Discussions stay factual because everyone analyzes the same traces.
What to Stop Doing
Avoid optimizing for public benchmarks that don’t match your particular task. Cease developing extensive research roadmaps without a clear plan for deployment. Don’t view prompts as magical fixes; instead, treat them as versioned and testable artifacts.
What to Start Doing
Begin each project by establishing an evaluator plan and defining what constitutes “true enough” and “safe enough. Work backwards from the customer, align your metrics to the desired outcome, and then iterate as you move along. Start with the smallest agent capable of moving a real metric, expanding the action space only when necessary. Ensure logging for explanations and audits is understandable by non-engineers. Regularly test for weaknesses by intentionally attempting to break the system.
Where This Is Going
The evaluator functions as reusable code, consisting of policy packs that ensure truthfulness, privacy, and compliance. Instead of a fixed setup, model portfolios allow independent swapping of primary and evaluator models based on cost, performance, and risk considerations. The system features self-healing agents that automatically de-rank, reindex, or quarantine sources flagged by evaluators for low faithfulness or security issues. AI is made attested and auditable by default, with signed artifacts for prompts, policies, tools, and indices, along with customer-visible trust indicators supported by trace links.
Small teams win because they align faster around outcomes, not papers. An embedded SME keeps “done” honest. A second model keeps the first model honest. And a Tier-1 deployment mindset, where you parallel-run until it becomes boring, keeps trust intact while you safely scale autonomy where it’s warranted.
By Joshua McDonald on October 16, 2025.
Exported from Medium on August 26, 2026.
Reader discussion