Architecting the Dual-Access Data Plane for BI and AI

How do we unify data access for both BI analysts and AI agents?

A single, coherent data plane must be engineered on a data lakehouse foundation, leveraging open table formats and a unified governance layer. The recent announcements from Google, Microsoft, and NVIDIA in early October 2026 are not isolated product releases; they are a clear signal that enterprise AI agents are moving from specialised, sandboxed applications to first-class consumers of core business data. This fundamentally breaks the legacy architectural pattern of maintaining separate data stacks for analytical BI and operational AI.

For years, we've architected a bifurcation: structured, governed data warehouses for BI dashboards, and separate, often ungoverned, data lakes or object stores for ML model training. This model is now obsolete. An AI agent tasked with summarising quarterly sales performance and drafting a customer email needs access to the same governed sales data as a CFO's Power BI dashboard. It also needs access to unstructured customer correspondence. Serving these two masters from separate, inconsistent systems is a recipe for governance failure and untrustworthy AI behaviour.

What is the foundational layer for this unified plane?

Open table formats like Apache Iceberg are the non-negotiable foundation, providing the transactional integrity, performance, and schema evolution required to reliably serve both workloads from a single copy of data. Legacy Hive-style tables, which lack atomic operations, create a brittle foundation where BI queries and AI data ingestion pipelines can easily corrupt data or produce inconsistent results. This is simply not tenable when an autonomous agent is making operational decisions based on that data.

Iceberg, particularly with the capabilities introduced in its V2 and V3 specifications, provides ACID transactions on petabyte-scale data lakes. This means an AI agent's feature engineering pipeline can write data atomically at the same time a critical financial report is being generated, without read/write conflicts. Features like partition evolution and schema evolution allow the platform to adapt to new data sources and changing business requirements without costly, high-risk data migrations—a critical capability for agile AI development. Furthermore, time travel and snapshot isolation provide an immutable audit log, which is essential for debugging AI behaviour and satisfying regulatory requirements.

How do we integrate unstructured and vector data without creating new silos?

By treating vector database indexes as a metadata layer over unstructured data residing in the lakehouse, rather than as a separate, disconnected system. The rapid adoption of Retrieval-Augmented Generation (RAG) patterns has led to a proliferation of vector databases like Pinecone, Weaviate, and pgvector. While powerful, they risk becoming the next generation of data silos, with duplicated data, divergent access controls, and no linkage back to the source of truth.

The correct architecture stores the raw, unstructured files—PDFs, meeting transcripts, support tickets—as objects in the lakehouse, catalogued by a system like Unity Catalog. The vector embedding pipeline reads these files, generates embeddings, and stores them in a vector database. Crucially, each vector entry must retain a foreign key or direct pointer back to the source object in the lakehouse. This allows an AI agent to perform a semantic search to find relevant documents, and then retrieve the original, full-fidelity documents from the governed lakehouse for its context window. This pattern ensures data is not duplicated, access can be governed centrally, and data lineage is preserved.

What role does the semantic layer play in an agentic future?

The semantic layer becomes the critical translation bridge, providing a consistent, governed vocabulary of business metrics and dimensions for both human analysts and AI agents to consume programmatically. It is the API for business logic. An AI agent can parse natural language, but it cannot intuit the precise, often nuanced, business definition of "Net Revenue Retention" or "Customer Lifetime Value."

Without a semantic layer, this logic is either embedded directly into brittle, complex prompts or hardcoded into the agent's application logic. This creates inconsistency; the definition used by the agent will inevitably diverge from the definition used in the BI tool. A centralised semantic layer, such as the dbt Semantic Layer or Cube, exposes these definitions via a stable GraphQL or JDBC interface. A BI tool connects to it to populate a dashboard. An AI agent connects to it to resolve a user query like "What was our monthly recurring revenue for the Hunter region last quarter?". The result is consistent and trustworthy, regardless of the consumer.

How does this architecture align with Australian AI governance?

This unified, metadata-driven architecture provides the traceability and auditable governance required by emerging regulations and frameworks like the NSW AI Assessment Framework (AIAF). The AIAF and related national policies place a strong emphasis on transparency, fairness, and accountability in AI systems. A fragmented, siloed data architecture makes demonstrating compliance a near-impossible forensic exercise.

By contrast, a unified plane built on Iceberg offers time travel, allowing auditors to reconstruct the exact state of the data an agent used to make a specific decision. A centralised governance tool like Unity Catalog provides a single place to manage and audit access controls for both human users and AI service principals. The semantic layer provides explicit, version-controlled definitions for the metrics that drive AI behaviour. This creates a transparent, auditable chain from raw data to business logic to agent action, which is fundamental to building responsible AI systems. For Sydney enterprises and public sector organisations, adopting such an architecture is not just a technical optimisation; it is a strategic imperative for managing regulatory risk. At Precision Data Partners, we architect these unified data platforms to help our clients de-risk their AI investments and build systems that are not only powerful but also compliant and trustworthy.