The Post-Unification Stack: Architecting Data Platforms for the Agentic Era

The longstanding architectural bifurcation between transactional (OLTP) and analytical (OLAP) systems is no longer fit for purpose. For decades, we engineered around this compromise: operational data in Postgres or Oracle, replicated via fragile ETL pipelines into a data warehouse or lakehouse for analytics. This introduced latency, complexity, and a fundamental disconnect between business operations and business intelligence. The emergence of agentic AI workloads has turned this architectural debt into an existential threat.

Databricks’ announcement of a "Lake Transactional/Analytical Processing" (LTAP) architecture at their summit yesterday isn't just marketing; it’s a formal recognition of this new reality. AI agents cannot operate on stale, 24-hour-old data. They require a unified view of the business, combining real-time transactional context with deep analytical history, to both perceive and act intelligently. The modern data platform must be redesigned from the ground up to support this unification.

The Transactional Lakehouse Foundation

The technical enabler for this unification is the maturation of open table formats. Formats like Apache Iceberg (v1.5.0) and Delta Lake (v3.2.0) provide the ACID guarantees and performance primitives that were once the exclusive domain of relational databases, but at petabyte scale. The ability to perform atomic multi-table transactions directly on the lakehouse, courtesy of specifications like Iceberg's REST Catalog and projects like Nessie, is a paradigm shift. We can now reliably land operational data directly into the lake, mutate it transactionally, and make it available for analytics in seconds, not hours.

Consider a classic e-commerce scenario. An agent needs to decide whether to offer a real-time discount to a user. This requires combining the user's clickstream events (analytical), their complete order history (analytical), and their current shopping cart contents (transactional). In the old model, the cart data lives in an OLTP database, and the historical data is in the lakehouse. The agent would require complex, high-latency federated queries. In an LTAP architecture, the `carts` table is an Iceberg or Delta table, updated with sub-second latency. The agent can query a single, transactionally consistent source for all required context, dramatically simplifying the logic and reducing decision latency.

A Dual Serving Layer: Semantic and Vectorial

With a unified data foundation, the next challenge is serving this data to agents with the requisite speed and in the correct format. Agents require two distinct types of context: structured, quantitative data for logical reasoning, and unstructured, semantic data for contextual understanding. This necessitates a dual serving layer architected for purpose.

First is the **Semantic Serving Layer**. This is where we serve metrics, dimensions, and low-latency aggregations. While querying the lakehouse directly with engines like Trino or Spark is suitable for ad-hoc analysis, it’s too slow for agentic loops. Instead, we materialise key datasets from our Iceberg/Delta tables into a real-time OLAP engine like StarRocks or Apache Druid. These systems can sustain sub-second p99 query latencies over trillions of rows, providing the performance envelope agents need. Critically, the definitions for these metrics are managed centrally in a metrics platform like the dbt Semantic Layer or Cube. This ensures that when an agent requests `customer_lifetime_value`, it receives a figure calculated with the same logic as the CEO’s dashboard.

Second is the **Vectorial Serving Layer**. For Retrieval-Augmented Generation (RAG), agents need to query for semantic similarity, not just structured values. This requires embedding unstructured and semi-structured data from the lakehouse—product descriptions, support tickets, customer reviews—into vector form. These embeddings are stored and indexed in a specialised vector database like Qdrant, Weaviate, or a Postgres instance running the pgvector (v0.7.2) extension. The key architectural challenge is maintaining synchronisation between the source of truth in the lakehouse and the vector index. Change Data Capture (CDC) streams from the transactional lakehouse tables, using tools like Debezium against Iceberg's change data feed, are the canonical pattern for ensuring the vector index reflects the latest state of the business.

Unifying Features and Metrics for Agentic Coherence

The distinction between a "feature" for a machine learning model and a "metric" for a business intelligence report is artificial and dangerous in an agentic world. An AI agent making a business decision needs to operate on a single, coherent set of definitions. When an agent assesses a customer's `is_high_churn_risk` feature, that feature must be derived from the same underlying data and logic as the `monthly_active_users` metric it sees in another context. Any divergence creates a schism in the agent’s understanding of reality, leading to erratic and untrustworthy behaviour.

This forces the convergence of the feature platform (e.g., Tecton, Feast) and the semantic layer. The modern stack needs a unified "context platform" where definitions are declared once and then materialised for different use cases. A definition for `active_customer` should be written once in a tool like dbt, then used to generate a feature in the low-latency feature store for real-time inference, a column in a BI dashboard, and a metric available to an agent via an API. This "define-once, use-everywhere" principle is non-negotiable for building reliable AI systems. Organisations that successfully implement this unified layer see dramatic improvements in both development velocity and operational stability.

The Control Plane: Data Contracts as Agentic APIs

The final architectural component addresses how agents *act* upon the world. Allowing an AI agent direct `UPDATE` or `DELETE` access to a lakehouse table is malpractice. Actions must be mediated, validated, and auditable. This is where the principles of data mesh, specifically the concept of data products and data contracts, become the control plane for agentic action.

Instead of modifying data, an agent interacts with a stable, versioned API exposed by a data product. For example, to cancel an order, the agent doesn’t execute `UPDATE orders SET status = 'CANCELLED' ...`. Instead, it makes a POST request to the `OrderDataProduct/v1/orders/[order_id]/cancel` endpoint. The data product's internal logic is responsible for executing the state change, validating business rules (e.g., order cannot be cancelled if already shipped), emitting domain events, and ensuring the underlying lakehouse table is updated atomically. The API specification, its schemas, latency guarantees, and quality metrics are all codified in a data contract. This approach transforms the data platform from a passive repository into an active, governed ecosystem where AI agents can participate safely and effectively.

By unifying the data foundation, building a dual serving layer for context, converging on a single source of business definitions, and implementing a contract-based control plane for action, we can construct a data architecture that is truly AI-native. This is the post-unification stack, and it is the necessary foundation for building the agent-driven enterprise of the next decade.