DuckDB and Parquet as the Agent Query Layer

Handing an AI agent a connection string to the production database is the fastest way to get it querying data, and the surest way to get wrong answers that look right. The appeal is obvious: no new infrastructure, no waiting on an engineering team, just credentials and a prompt. But the output this setup produces is unreliable in a way that has nothing to do with how capable the agent is. Two agents asked the same question about revenue have no guarantee of returning the same number, because neither one is working from a shared definition of what "revenue" means in that business.

The damage isn't limited to inconsistency between agents. Production databases built on Postgres or similar systems are row-oriented, built to handle many small concurrent writes with strict transactional guarantees. The SQL it generates, the joins it works out, the business logic it reconstructs to figure out what "active customer" or "churned account" means, all of it lives inside that one conversation and disappears when the session ends. The next agent that gets asked the same question starts over, rediscovering the schema, re-deriving the same definitions, writing SQL that may or may not agree with what came before. Large result sets pulled directly into an agent's context window add cost without adding accuracy, since raw rows don't carry the interpretive logic needed to turn them into a correct answer.

Agents aren't careless. The interface they're given is wrong for the job. A typical analytical request forces an agent through the same sequence every time: discover the schema, identify the relevant tables, generate SQL, load the results into context, reconstruct the business definitions needed to interpret those results, then calculate whatever metric was actually asked for. Nothing persists, nothing accumulates, and nothing gets more reliable with repetition. A more capable model pointed at an unmapped schema still has to invent its own definition of "active user" or "net revenue," and it will invent one independently every single time, because nothing in the architecture gives it a definition to inherit. The fix has to happen upstream of the model, in the interface the agent is given to work with.

The flaw runs deeper than any single agent's reasoning ability. Without a governed interface standing between agents and the production schema, every session reconstructs its own version of the business logic, and identical requests end up with different answers depending on which agent handled them and when. That's a structural gap, and it's why separation of concerns matters here: platforms like Dreambase pre-materialize refined datasets as governed Parquet files, so agents work from the same analytical surface every time instead of negotiating directly with raw production tables.

DuckDB and Parquet's contributions to an agent query layer

DuckDB and Parquet each solve one half of the problem described above, and the combination is what closes it. Parquet gives agents a safe, pre-built surface to read from. DuckDB gives them an engine that can scan that surface without needing a server, a pipeline, or a live line into the production database.

Parquet is a columnar storage format that organizes data by column rather than by row, which is exactly the layout analytical queries want. The file is a snapshot taken at a point in time, not an open connection into live operational data, and an agent querying that snapshot can't reach anything beyond what was captured in it.

DuckDB is the other half. For an agent running an exploratory scan before narrowing down to a specific question, that kind of fast, local, bounded querying is close to the ideal interaction model: the dataset is well-defined, the engine is immediately available, and nothing about the exploration touches a system anyone else depends on for uptime.

Put the two together: refined datasets get materialized as Parquet on a schedule, and DuckDB is the engine that queries them. Neither piece does this job alone. A Parquet file with no engine attached is just a file an agent has no way to reason over. DuckDB pointed directly at a production database undoes the entire benefit, because now it's running analytical scans against an OLTP system that was never built to absorb them. Parquet supplies the bounded, read-only snapshot; DuckDB supplies the engine that can query it without a round trip to production. Dreambase implements this pairing directly, serving pre-calculated datasets as Parquet files that are queryable through DuckDB via a single MCP server, so agents never touch production while still getting fast, accurate analytical context.

Governance before performance

The instinct is to read DuckDB and Parquet as a speed story, faster queries, lower latency, cheaper compute, but the real value is in what they're allowed to see and trust. Routing agents through pre-materialized Parquet queried by DuckDB is, before anything else, a decision about what data an agent is allowed to see and what it's allowed to trust.

A Parquet dataset can carry things a live production table never will: a stable name and a clear statement of purpose, context about where the data came from and how it relates to other datasets, defined dimensions and measures, explicit metric and KPI definitions, a record of when it was last refreshed, quality and profiling metadata, permissions and usage rules, and a lineage trail showing how it was derived. None of that lives inside a raw Postgres table. It has to be built, attached, and maintained deliberately, and once it exists, the dataset becomes useful to far more than the single workflow that originally created it.

The governance boundary sits between raw source data and the refined analytical product, and agents should operate only on the refined side of that line. Because the Parquet file is materialized before an agent ever queries it, row-level security and scoped access can be enforced at the point the dataset is built. Row-level security enforced at build time means an agent can only ever see the rows that were written into the file. Supabase is moving in the same direction at the platform level: new tables created in the public schema will no longer be exposed to the Supabase Data API by default, a change enforced on all projects by October 30, 2026. The direction is narrower exposure by default, and agent access patterns built on top of Supabase or any other operational database should follow that same instinct.

Their finding was not that giving agents raw retrieval access to a large corpus of prior queries improved accuracy; by their own account, that approach moved accuracy by less than a point. Snowflake's own benchmarks point the same direction, showing that grounding a model in a governed semantic model lifts text-to-SQL accuracy past 90%, while letting a model auto-generate its own definitions from raw tables just reproduces the ambiguity that governance was supposed to remove.

Fast queries against ungoverned data still produce wrong answers quickly. Governed data with no fast way to query it is correct but unusable at the speed agents need to operate. The architecture earns its place in an agentic stack because it delivers both at once: Parquet and DuckDB supply the speed, and the discipline of materializing a governed, well-defined dataset before any agent touches it supplies the trust. Neither half substitutes for the other.

The agentic data factory pipeline: from raw Postgres to governed Parquet

The right way to think about Postgres, including a Supabase project, in an agentic stack is as the system of record, not the system agents query directly. A downstream layer of governed Parquet datasets, queried through DuckDB, is the actual interface agents use. Postgres stays row-oriented, built for high-concurrency writes and strict ACID guarantees, exactly the job it should keep doing. Large analytical scans across millions of rows don't belong there, because they degrade the latency of the application workloads the database exists to serve.

The pipeline that gets data from that operational source into an agent-ready dataset runs through several distinct stages. From there it moves into schema and relationship discovery, figuring out what tables exist and how they connect. The final stage is where dashboards, reports, APIs, and agents connected over MCP actually consume the result.

A concrete example makes the shape of this clearer. The source systems, Supabase and Stripe, remain responsible for operational truth: new accounts, new charges, the actual transactions as they happen. The refined dataset is the reusable analytical interface built on top of that truth, and it's what an agent actually queries when someone asks how revenue health is trending.

Not every dataset built this way deserves to live forever, and the pipeline accounts for that with three lifecycle states. Work that turns out to matter survives past the conversation that produced it; one-off exploration doesn't accumulate as clutter that nobody maintains.

For teams already running on Supabase, there's a practical on-ramp into this pattern that doesn't require standing up a separate warehouse: pg_duckdb, which accelerates analytical queries running directly on Postgres. Alongside that, access discipline matters as much as the pipeline itself: agent connections should default to read-only, routed through narrow views or approved tools rather than raw tables, with parameters validated, queries logged, destructive operations gated behind human approval, and row-level security or scoped service roles enforced at the data layer rather than left to the model's memory of what it's allowed to touch.

Pre-calculating metrics so agents receive governed answers rather than rebuilding KPI logic independently

Metric logic doesn't belong inside an agent's reasoning process, reconstructed fresh on every task. It belongs stored alongside its definition, as a governed product any agent can read directly without re-deriving the formula itself. Without that governance, three separate agents asked to report Monthly Recurring Revenue will write three separate SQL statements, each reflecting a slightly different assumption about what counts, and produce three different numbers with no way to tell which one, if any, is right.

The fix is pre-calculation paired with versioned definitions. A data factory built this way calculates and stores metrics ahead of time, along with their written definitions, their current values, their historical values, day-over-day changes, and rolling comparisons across 7-day, 14-day, and 30-day windows, plus any related business events and anomaly flags. The exact storage format can vary from one setup to another, but the underlying principle holds regardless: the metric gets computed once, under one definition, and every agent that asks for it afterward reads the same answer.

A concrete example shows what this looks like in practice. An agent that reads this object gets a complete, context-rich answer. It isn't handed a raw number stripped of the logic behind it, left to guess at what "net revenue" was supposed to mean in this business.

Anthropic's own experience inside their data team shows why this has to be built deliberately. A single definition, shared identically across a founder, a board, and every agent that queries the data, does more operational work than any dashboard feature could, because it settles the "which number do we use" argument before anyone has the chance to start it.

How MCP exposes the governed data layer to agents

None of the governance work described above matters if agents have no efficient way to reach it, and that's the role the Model Context Protocol plays. MCP is an open standard for connecting AI models and agents to external tools and data sources through one consistent interface, instead of building a custom integration for every system an agent might need to touch. MCP is the delivery mechanism, not the intelligence and not the governance. An agent connected over MCP to ungoverned, undefined tables will still produce confidently wrong answers, because MCP moves data, it doesn't clean it or define it.

A specification revision dated July 28, 2026 made MCP stateless at the protocol layer. After it, that same server can run behind a plain round-robin load balancer with no session affinity required, which matters directly for deploying a governed data layer to fleets of agents running in parallel, since none of them need to be pinned to a particular server instance to keep working correctly.

The layering that makes this architecture coherent places MCP in front of the governed Parquet datasets and the DuckDB engine that queries them. MCP's job is moving pre-modeled data and query results out to agents. It never touches the production database directly, because by the time MCP is involved, the data has already been materialized, cleaned, and governed upstream. A single MCP server exposing every governed dataset is a meaningful simplification on its own: agents working in different environments, whether that's Claude Code, Codex, or a custom pipeline built in-house, all reach the same pre-calculated metrics and modeled datasets through the same interface. The "one definition" guarantee holds regardless of which agent happens to be asking: every agent is reading from the same governed layer.

Isolation at the fleet level is worth a brief mention too. Write access is handled separately from all of this, and deliberately so. The MCP layer serving analytical reads should stay strictly read-only.

Dreambase implements this pattern directly for Supabase teams and others: agents query modeled Parquet on DuckDB, never production, through one MCP server that drops into any workflow already in use. What makes that MCP connection trustworthy is that the governed dataset layer sitting behind it was built correctly from the start.