Architecting the Compound AI Execution Fabric

The architectural patterns that brought us through the first wave of generative AI are already obsolete. Serving a single foundation model behind an API endpoint was a solved problem by 2024. Today, the critical challenge is serving the complex, multi-step, and often multi-modal workflows of production agentic AI. The monolithic inference stack, optimised for single-shot request-response, is a bottleneck that actively degrades performance and inflates cost when faced with these new workloads.

Recent industry movements, such as the 1.0 release of the `llm-d` distributed inference project, confirm the trend: serious engineering effort is shifting from the model serving engine to the broader execution environment. We are entering the era of the Compound AI Execution Fabric—a necessary evolution for any organisation serious about deploying agents at scale.

Why are traditional inference stacks failing agentic workloads?

They are failing because they were designed for a different paradigm. Traditional inference servers like early versions of TGI or vLLM are optimised for stateless, single-model throughput, whereas agentic systems are inherently stateful, multi-turn, and increasingly multi-model.

This mismatch creates several acute problems. The most significant is ‘compound latency’. An agentic task that requires five sequential model calls—for planning, tool selection, data retrieval, synthesis, and final response generation—doesn't just sum the latency of each call. The p99 latency of the entire sequence becomes catastrophically high, as any single slow response delays the entire chain. A system where each step has a p99 latency of 2 seconds does not yield a 10-second p99 overall; the tail-end probabilities multiply, resulting in an unusable user experience.

Furthermore, these stacks struggle with state management, particularly the Key-Value (KV) cache. In a multi-turn conversation, preserving the KV cache between calls is crucial for performance. In an agentic workflow, this context must be managed across different models and tool invocations, a task for which monolithic servers are ill-equipped. Loading a large planning model, then a smaller tool-using model, then an embedding model, forces inefficient memory swapping and cache invalidation, destroying throughput.

What defines a Compound AI Execution Fabric?

A Compound AI Execution Fabric is a distributed, heterogeneous system that separates architectural concerns: request routing, multi-step orchestration, state management, and decentralised model execution. It recognises that the ‘intelligence’ of the system lies in the orchestration, not just the model.

This fabric is composed of four key components:

1. **Dynamic Routing Layer:** This acts as a smart load balancer for AI requests. It decouples the client from the specific inference server or model. A request to generate a financial report summary could be dynamically routed to a fine-tuned Llama-3-70B model running on an A100 cluster, while a subsequent request to extract figures might be sent to a smaller, quantised function-calling model on an L4 GPU. This layer makes decisions based on payload, latency requirements, and real-time hardware availability.

2. **Workflow Orchestration Engine:** This is the brain of the agentic system. It manages the directed acyclic graph (DAG) of the agent's task. It executes a step, parses the output, calls a tool (e.g., an API, a database query), and then routes the result to the next model in the chain. Frameworks like LangGraph or custom-built state machines live here, entirely separate from the LLM inference servers themselves.

3. **Distributed Inference Core:** This is the muscle, comprising a heterogeneous fleet of GPUs and serving engines (vLLM, TensorRT-LLM, SGLang). Leveraging standards like the `llm-d` project allows for tensor parallelism across nodes for massive models and efficient batching for smaller ones. The key is that this layer simply executes model forward passes as instructed by the orchestrator; it contains no business logic.

4. **Externalised State Management:** To solve the KV cache problem, its state must be moved off-GPU and out-of-process into a dedicated, high-speed cache like Redis or a specialised vector-memory service. This allows an agent's conversational context to persist across calls to different models on different nodes, drastically reducing the re-computation of prompt tokens and improving end-to-end latency for stateful interactions.

How do you optimise for cost and throughput in this new fabric?

Optimisation moves beyond single-model quantisation to a holistic, fabric-wide strategy involving model cascading, aggressive hardware tiering, and advanced serving techniques. The goal is to use the least expensive compute necessary for each step of a workflow.

Model cascading is a primary strategy. A cheap, fast model (e.g., a quantised 8B parameter model) first classifies an incoming task. If it's a simple query, it answers directly. If it requires complex reasoning, it escalates the task to an expensive, frontier model. This simple routing logic, implemented in the orchestration layer, can slash costs by ensuring high-capability models are used only when essential.

This is coupled with hardware tiering. The fabric should manage a mixed-hardware environment: H200s for training and frontier model inference, A100s for workhorse 70B models, and L4/T4s for smaller models, embedding, and classification tasks. The router's job is to match the task to the right-sized hardware.

Finally, advanced techniques like speculative decoding and efficient quantisation (e.g., AWQ, fp8) are applied across the fleet. Speculative decoding, where a small 'draft' model generates tokens that a larger model merely verifies, can triple throughput. In a compound fabric, the orchestrator can even use different models for drafting and verification, further optimising the process.

What does this mean for Australian organisations?

For Australian organisations, particularly in regulated sectors like finance and government, this architectural shift provides critical control points for implementing robust governance and aligning with local standards like the NSW AI Assessment Framework (AIAF).

The decoupled routing and orchestration layers are natural enforcement points for AI governance and compliance. Policies can be implemented directly into the fabric. For example, a rule in the router can prevent any request containing personally identifiable information from being sent to a model hosted outside of an Australian data centre. An orchestration rule can enforce a human-in-the-loop review step for any agent action that involves a financial transaction over a certain threshold.

This architecture provides the deep traceability required by the NSW AIAF's principles of transparency and accountability. The orchestration engine naturally produces a complete execution trace for every agentic workflow: which models were called, with what inputs, what tools were executed, and what the intermediate results were. This creates a verifiable audit trail that is simply not available from a black-box, single-model endpoint. It moves organisations from simply trusting a model's output to understanding its entire decision-making process.

At Precision Data Partners, we architect these compound execution fabrics to provide exactly this level of control and transparency. The ability to build a hybrid fabric—routing sensitive workflows to sovereign models hosted locally while leveraging global frontier models for less sensitive tasks—is how NSW organisations can balance world-class capability with stringent data residency and regulatory requirements.