The era of monolithic data warehouses is over; AI agents demand a polyglot query plane that combines lakehouse scale with interactive, exploratory data
What is the 'agentic query pattern' and why does it break traditional architectures?
The agentic query pattern is a new class of workload characterised by a high volume of small, exploratory, and iterative queries generated by autonomous AI systems. This pattern cripples traditional, monolithic data warehouses that were engineered for large, infrequent batch queries driven by human analysts.
For two decades, we optimised data platforms for a specific paradigm: a human analyst runs a handful of complex, resource-intensive queries to populate a dashboard that will be viewed by hundreds. The economic and performance trade-offs of systems like Snowflake, Redshift, and BigQuery are built on this assumption. The workload is predictable, chunky, and latency tolerance is measured in seconds or minutes.
An agentic AI system operates on entirely different principles. A financial analysis agent, tasked with investigating an anomaly, will not issue a single, monolithic query. Instead, it will execute a rapid-fire sequence of dozens, or even hundreds, of small queries: a `SELECT COUNT(*)` to gauge table size, a `SELECT DISTINCT [category]` to understand cardinality, followed by a series of targeted `SELECT` statements with varying `WHERE` clauses to test hypotheses. Each query is a 'poke' at the data, informing the next action in a complex reasoning loop. This is an interactive, conversational pattern executed at machine speed.
The data architect's role is no longer about building a single, perfect highway for data. It's about designing a city grid with different road types—from multi-lane motorways to local laneways—and implementing the traffic signals that route each vehicle to its destination most efficiently.
AWS's acquisition of DuckLabs, the company behind DuckDB, on 31 August 2026 is the most significant industry validation of this shift. In their announcement, AWS explicitly cited DuckDB's suitability for AI agents that "poke and experiment their way through data". A heavyweight cloud provider acquiring a lightweight, in-process analytical engine is a clear signal that the monolithic model is insufficient for the AI-native future.
How do you build a data plane to serve this new pattern?
The solution is not to replace the data lakehouse but to augment it with a federated, multi-engine query plane that intelligently routes requests to the most appropriate engine. This polyglot architecture optimises for latency, cost, and complexity, treating the underlying query engines as a portfolio of specialised tools rather than a single silver bullet.
This modern data plane consists of three primary components sitting on top of an open-format foundation:
1. **The Foundation:** The data lakehouse, built on open table formats like Apache Iceberg or Delta Lake, remains the single source of truth. It provides the scale, reliability, and transactionality required for enterprise data. A governance layer like Databricks Unity Catalog provides a unified namespace, access control, and lineage across all data artefacts.
2. **The High-Throughput Engine:** A distributed processing engine like Apache Spark or Trino handles the traditional, large-scale workloads. This is for your heavy ETL/ELT, batch model training, and large-scale BI aggregations. It’s the multi-lane motorway in our city grid analogy—powerful, but with high startup costs and not suited for short trips.
3. **The Interactive Engines:** This is the critical new layer. It includes real-time OLAP databases like Apache Druid or ClickHouse for sub-second querying on pre-aggregated, denormalised datasets. Crucially, it also includes embedded, in-process engines like DuckDB. An agent can spin up a DuckDB instance within its own execution environment to query a subset of Parquet files directly on object storage, achieving millisecond-level latency without the overhead of connecting to a distributed cluster.
What does this mean for Australian organisations?
Australian organisations, particularly in regulated sectors, must architect these new data planes with robust governance and compliance at their core. The proliferation of agent-driven queries creates a new and complex audit surface that demands a proactive approach to AI governance, aligning with emerging standards like the NSW AI Assessment Framework (AIAF).
The AIAF, and similar global frameworks, require transparency, accountability, and contestability in automated systems. When an AI agent makes a decision based on data, you must be able to trace that decision back through the entire query chain. This is impossible if you have agents randomly firing queries across a fragmented landscape of data silos. A unified governance layer, such as Unity Catalog, becomes non-negotiable. It must capture the lineage of every query, regardless of which engine executed it, and link it back to the specific agent and workflow that initiated it. This provides the audit trail necessary to satisfy regulators and build trust in your AI systems.
For enterprises across NSW, from financial services in Sydney to the growing technology sector in the Hunter, this represents a significant architectural challenge. The temptation is to allow teams to spin up their own specialised engines, but this leads to governance chaos. The mandate for data architects is to design a centrally governed platform that *enables* the use of these diverse tools within a secure and auditable framework.
How should data teams evolve their skills and processes?
Data teams must transition from being builders of static pipelines and dashboards to becoming managers of a dynamic, multi-engine data access platform. This requires a fundamental shift in focus towards platform engineering, cost optimisation, and the enforcement of data contracts.
The data engineer of yesterday was a Spark and SQL expert, focused on optimising batch jobs. The data engineer of tomorrow is a platform architect, skilled in Kubernetes, service meshes, and query routing logic. Their primary task is not to write the queries, but to build the platform that allows agents to write and execute queries safely, efficiently, and cost-effectively. Performance optimisation is no longer about tuning a single Spark job; it's about ensuring a query from a low-latency agent is routed to DuckDB, while a massive aggregation is routed to Spark, all seamlessly.
Your immediate priority is not to replace your lakehouse. It is to build a low-latency, exploratory sidecar capability and begin routing a small subset of agentic workloads to it. Measure the performance and cost differences obsessively.
This new operating model elevates the importance of data contracts. When AI agents are your primary data consumers, schemas, data quality metrics, and semantic definitions can no longer be informal agreements. They must be machine-readable, version-controlled artefacts that are automatically enforced at the platform level. At Precision Data Partners, we work with clients to design these federated data platforms, ensuring they deliver the performance AI demands without sacrificing the governance and control that enterprises require.
See how this applies in practice on our Retail solutions page.
Ready to apply these patterns in your stack?
Book a free 45-minute AI readiness call with the Precision Data Partners team.
Book a Free Audit
