Keep the bucket. Add a control plane.
VGI puts a SQL front door on the data you already store. Different rows and columns per customer, a log of every query, and functions you can charge for — without moving a byte out of S3.
Where you are now
Right now, your product is a folder and a set of credentials
You already sell or provision access to data. In practice that means Parquet in a bucket, a prefix per customer, and a credential handed out at signup. It works — it's why you haven't replaced it — but the shape of it decides what you're able to sell. A file is the smallest unit of access, so every entitlement is a copy, every tier is a pipeline, and every question your customer asks about the data is a support ticket.
What you can't do is the interesting part. You can't see which questions they ask. You can't ship a column to one tier and not another. You can't reorganize your own storage without emailing forty accounts. And when a contract ends, the last copy is still theirs.
VGI doesn't replace any of that. It puts a control plane in front of it: the bucket stays, the bytes keep flowing straight out of it, and what changes is that every read becomes a decision you make — at query time, per caller, on the record.
What you're selling
A folder can do one thing. A service can do several
A file has exactly one behaviour: it can be fetched, whole, by anyone holding the credential. Everything a data vendor wishes it could do — return only this customer's rows, describe itself, run something — isn't a missing feature of files. It's a thing files can't have, because nothing is running.
Put a service in front of the same bucket and those all arrive together. They aren't tiers you climb or sell in sequence; they're what falls out of there being a process on the other end of the request.
| What a customer wants | A folder in a bucket | A service in front of it |
|---|---|---|
| Get the raw data | The whole folder, identical for every customer. | The same files, read straight from your bucket at full speed — with a scoped, short-lived credential instead of a permanent one. |
| Get only their rows | Pre-materialize a copy per customer, per tier, and backfill it whenever an entitlement changes. | One table. The caller's identity decides the rows, on every query. |
| Find out what's in it | Read your documentation, or ask you. Your directory naming convention is the API. | DESCRIBE it. The schema, the columns, and the types answer for themselves. |
| Use your methodology | Not possible — you would have to ship them the code. | Call it as a function. The scorer, the matcher, the model stays on your infrastructure. |
The first row is the one that surprises people: a service can still hand back the raw files, at full speed, straight from your bucket. That's direct read, and it's what makes this an addition to what you have rather than a replacement for it.
The one worth the most, though, is your own code. Rows are a commodity — someone will always undercut you on rows. A matcher, a geocoder, a scorer, a model: that's methodology, it's what customers actually pay a premium for, and until now you couldn't sell it without shipping it. As a function on your worker it stays yours, runs on your infrastructure, and gets called by name.
Not a new idea, exactly — Query Farm's own Airport extension proposed it first, over Arrow Flight. VGI takes the premise further: any language can implement a worker, a worker runs over a local pipe as easily as HTTP, and a single query can join across more than one worker at once.
The part people don't expect
Your worker doesn't move the bytes
The obvious objection to putting a SQL layer in front of your data is that everything your customers read now flows through a server you run and pay for. It doesn't have to. When the data is already Parquet, Iceberg, or Delta in object storage, the worker can answer with a plan instead of rows — which files, and a short-lived credential to read them — and the customer's DuckDB fetches them itself.
Three things follow from that. You don't pay to move the bytes twice — no egress to your server and then out again, just the one hop you already pay for today, and on R2 that hop is free. Your customers read at object-storage speed, so you are not a bottleneck you have to capacity-plan. And your control plane stops being sized by your data volume: a worker that answers which-files-for-whom does the same work whether the table behind it holds a gigabyte or a petabyte, because it never touches either.
Direct read enforces at file granularity, not
row granularity. The customer's DuckDB ends up holding your file list and a
credential scoped to those files — so whatever you can express as a prefix scope is
enforced, and anything finer is not. A per-caller WHERE clause
in the plan you hand back runs on their machine, which is fine for an honest query and
worth nothing against a determined one.
So it's a per-table decision, and most vendors end up running both.
Bulk tables go direct — the whole
dataset, or a tier whose prefix already is the entitlement boundary — at object
storage speed. Entitled tables stream through
the worker, where rows and columns are decided per caller and every read is
yours to log. Same interface, same ATTACH; you choose per
table.
Two consequences worth knowing before you design around it. On the direct path your log records what you authorized, not what was read. And revocation is bounded by the credential's own lifetime rather than instant — the extension never serves a token past its expiry, so the window is short, but it isn't zero.
One table, many places
A table doesn't have to live in one place either. Last week's rows in your operational store, last quarter's in Parquet on S3, everything older in cold archival storage — one table to the customer. The worker declares which source covers which range, and a query that only asks for recent rows never touches the other two.
The business version of that: your storage economics stop being a customer-facing contract. With files, moving last year's data to colder storage is a migration email to forty accounts, all of whom hardcoded your paths. With a catalog, it's a decision you make on a Tuesday and nobody downstream notices.
Worth knowing before you build on it: multi-source tables are read paths.
INSERT routes to a single nominated source, and
UPDATE, DELETE, and time travel are
refused rather than half-supported. Sources are expected not to overlap — if two claim the
same rows you get duplicates — obvious in a row count, invisible inside a
SUM() — so keeping the ranges apart is on you.
The mechanism
Your consumers, your services, one SQL layer between them
SQL clients, Excel, ODBC drivers, and browsers can all already talk to DuckDB directly —
that's DuckDB's own client ecosystem, and none of it is VGI-specific. VGI is strictly the
other leg: DuckDB reaching your services. ATTACH a worker and
query it like any other catalog — the same pattern whether it hands back rows, a scalar
function, or a live-refreshing feed.
“Your access control policies” is deliberately generic — a worker can implement its own, or sit behind Row & Column Security, a Query Farm product (hosted or on-prem) that enforces row- and column-level security in front of any VGI worker without the worker having to implement any of it itself.
Underneath, DuckDB talks to the worker over VGI-RPC — Apache Arrow IPC, carried over a local pipe or plain HTTP. The worker runs outside DuckDB's own process, so it's never tied to the extension ABI that forces a native extension to recompile on every DuckDB release.
More on the protocol itself: Architecture →
In practice
Same data, different access for different people
Say you're a data vendor — earthquake data, for instance. A subscriber in California
doesn't need Alaska's rows, and shouldn't be able to query outside what they bought. With
access tied to who's asking, the same earthquakes table serves a
California-only subscription and an Alaska-only subscription — same schema, same SQL,
different rows, enforced on every query, not just checked once at signup.
None of this is limited to a single source, either. Because DuckDB and VGI speak the same SQL regardless of where a table actually lives, a query can join your own data against a customer's, or against a partner's — the benefit of SQL over a data dump is that it composes.
The other side of the boundary
What your customers install
DuckDB, and one extension. That's the whole ask.
INSTALL vgi FROM community;
LOAD vgi;
ATTACH 'eq' (TYPE vgi, LOCATION 'https://data.yourcompany.com/', bearer_token '$TOKEN');DuckDB is the reason this is a small ask rather than a platform decision. It's a single embedded binary with no server to stand up, and it connects to whatever your customer already works in — Python, R, Java, Node, Go, Rust, the CLI, a notebook, the browser, or anything speaking ODBC and ADBC. Wherever they can already read your Parquet, they can run DuckDB, and your catalog shows up as ordinary SQL objects inside it.
And they don't have to. Customers who can't or won't add a dependency keep pulling files from the bucket exactly as they do now — that path never goes away, and direct read means the ones who do adopt are reading the same objects anyway.
The commercial part
What you can charge for
With files, a tier is a pipeline, so most vendors ship two or three of them. When a tier is a policy branch instead, the arithmetic changes — and the money usually isn't in the premium tier. It's in the narrow, cheap tier you can't afford to build today.
Rows as a price axis
Geography, sector, ticker set, time window. A tier stops being a pipeline and becomes a policy branch, so shipping twenty of them costs about what shipping three costs today.
Columns as a price axis
Base columns in the standard tier, enriched or scored columns above it. Your sample dataset stops being a separate artifact you maintain and becomes the same table with fewer columns.
Freshness tiers
The same table at T+0, T+1, or T+30. The implementation is a filter on a timestamp — this is the cheapest new SKU on the list, and the one file vendors already fake with a delayed bucket.
Compute as a product
The matcher, the geocoder, the scorer, the model. A function call is a discrete, priceable unit, and it is the one thing on this page that files structurally cannot carry.
Bundled joins
Charge for the composite — your data joined against a partner's, delivered as one table, with neither side shipping the other a copy. With files that needs a third party to do ETL, and that third party takes the margin.
Trials that expire on their own
A 14-day, three-column, one-region trial is a policy. With files a trial is a permanent copy, which is why file vendors under-offer trials and starve their own funnel.
VGI does not meter or bill anyone today. The per-query decision log comes from row & column security, which is a separate product we operate rather than part of the extension — and that log is an input to a meter. It is not a meter, a quota, a rate limit, or an invoice. If your pricing depends on counting, you are building the counter.
Two related honesties. A query is a poor billing unit — one that scans ten rows and one that scans ten billion are both one query — so rows or bytes are more defensible and less predictable for the customer. And usage pricing is a sales problem before it's a technical one: plenty of procurement departments would rather have a fixed line item.
The other direction
The same thing, pointed inward
Everything above describes selling to customers. Point it at your own company and nothing about the mechanism changes: one SQL surface over finance and sales, where what a person sees depends on their region or job function rather than which export they were sent.
The connection is worth stating plainly, because it runs in both directions. A platform team that unifies internal sources behind one SQL surface has already built the thing you would need to sell data to a partner — same worker, same policy engine, same log. The only difference is whether the caller's identity resolves to an employee in an SSO group or to a customer account. Most companies build the internal one first and find out later that it was the pilot.
However you want to run it
Where the worker runs
Selling access changes where a worker lives. A local subprocess serves one DuckDB session
on one machine; a subscriber base needs a long-lived HTTP service that any session can
reach — and then someone has to operate it, either you or Orchard. Either way the consumer
side is unchanged: one ATTACH, and the only thing that differs is
the LOCATION.
That choice — where a worker runs, and every LOCATION form
DuckDB can connect through — is covered in full here:
What it costs you
What you actually have to run
A worker is a normal service in a normal language — Python, TypeScript, Go, Rust, or Java — so “we need a source nobody has built a worker for” is an afternoon, not a blocker. Run it on your own infrastructure and nothing touches Query Farm at all, or let Orchard run it for you.
Filter pushdown
On the streamed path a WHERE clause reaches the source before a byte crosses the wire, so a careless query doesn't become your egress bill — and a worker can refuse an unfiltered scan outright. On the direct path the filter runs in the customer's DuckDB, so it stops an honest mistake at plan time and nothing more.
Result caching
Client-side and TTL-based, surviving across queries and worker restarts. A worker declares how fresh its data needs to be, and repeats inside that window never reach you.
Auth you don't write
ATTACH carries a bearer token or a full OAuth refresh-token flow. The extension handles it and redacts tokens from telemetry; you don't hand-roll auth per worker.
-- A static token, handled by the protocol itself
ATTACH 'crm' (TYPE vgi, LOCATION 'https://crm.example.com', bearer_token 'sk_live_...');
-- Or a full OAuth refresh-token flow — no per-worker auth code required
ATTACH 'crm' (TYPE vgi, LOCATION 'https://crm.example.com', oauth_refresh_token '...');On licensing. VGI is source-available, not open source. The restriction exists to stop someone repackaging VGI itself as a competing gateway product — not to stop you selling data through it, which is what this page is about. The clause is written broadly, so if you're building something where the line isn't obvious, ask us and we'll put the answer in writing.
Row- and column-level enforcement is a separate hop we operate — hosted or on-prem — that doesn't trust the worker behind it. Row & column security → Workers other people have already built and hosted live in Orchard.
Not a screenshot
See it live, right now
Weather, earthquakes, a self-refreshing train departure board, a map of real locations — every example on the VGI tour runs against a real, hosted worker, live, in your browser. We could have faked that demo. We didn't.
In context
What changes
Sentences you have probably said out loud in the last month, and what replaces each one. The middle column matters most: none of these is something a better naming convention fixes, because a file is the smallest unit of access there is.
| What today looks like | Why files can't fix it | What replaces it |
|---|---|---|
| “Adding a customer means adding a bucket prefix and a pipeline.” | A file is the smallest unit of access, so every entitlement has to be pre-materialized. N customers x M tiers is a copy explosion, and changing one is a backfill. | One table, one policy set. The caller's identity selects the rows. A new customer is a policy branch, not a pipeline. |
| “Half my support tickets are “which file do I read?”” | Your directory naming convention is your public API, and it is undocumented. Every customer writes their own loader against it. | SELECT * FROM eq.public.earthquakes. The schema is self-describing, and your data dictionary becomes DESCRIBE. |
| “I have no idea what they actually query.” | Bucket access logs record byte ranges. They do not record questions. | One log line per query, including which filter was applied and which columns were withheld. You find out what your customers actually ask. |
| “I can't charge for usage, because I can't see usage.” | All-you-can-eat is a property of files, not a pricing decision. One SKU: an annual dataset license. | A per-query event stream you can price against — though see the note above: that is an input to a meter, not a meter. |
| “There is nowhere to put the premium thing.” | A bucket holds files. A file is a file. There is no surface to merchandise on. | Schemas and functions. A premium schema sits beside the standard one; a scoring function is a SKU with its own price. |
| “I can't reorganize storage without emailing forty accounts.” | Customers hardcode paths, so your physical layout became a customer-facing contract the day you shipped it. | The catalog is the contract. Repartition or tier to cold storage and the next query picks up the new layout — nobody downstream rewrites a path. (On the direct path your object keys do reach the client, so treat them as visible, not as an interface.) |
| “Adding a column breaks half of them and the other half never notice.” | There is no deprecation channel and no per-tier projection — everyone gets the same files. | Columns are projected per caller. Ship a column to one tier before the rest, and deprecate by policy instead of by email. |
| “When a contract ends, they still have everything.” | Revoking a copy is a legal instrument, not a technical one. | Access ends on the next query — or, on the direct path, when the credential expires minutes later. Honestly: they keep whatever they already read. Nothing changes that. |
The other side of it
When a bucket is the right answer
Some datasets don't need any of this.
If your data is small, static, and identical for every customer, a bucket and a signed URL is the correct architecture. Don't add a control plane to it.
If every one of your customers already lives in the same warehouse, use that warehouse's sharing. It will be less work and they will be happier.
And if you have one customer, you don't have an entitlements problem yet. You have a customer. Come back when you have twelve.
Questions
FAQ
How is this different from Delta Sharing or an Iceberg catalog?
Those grant access to a table by handing over the table. What VGI takes away isn’t the bytes — with direct read the bytes still come straight from your bucket — it’s unconditional, permanent, unaudited access. Rows and columns are decided per caller on every query, entitlements change by editing a policy rather than rebuilding a share, and access ends on the next query — or, where a credential was vended for a direct read, when that credential expires. What a customer already read, they keep; nothing changes that, in any system.
Is this actually live, or is it cached?
Both, by design. Every query is live by default — there’s no batch sync to fall behind. When a worker advertises a freshness window (the live train departures example uses 30 seconds), VGI’s client-side result cache can serve a repeat query from that window without hitting the source again, so you get freshness without hammering the API underneath it.
What about rate limits and authentication?
ATTACH itself carries a bearer token or a full OAuth refresh-token flow — credentials are handled by the protocol, not hand-rolled per worker. Rate limiting is up to the source; result caching and filter pushdown both reduce how often a worker has to ask it for anything.
Can our data stay entirely on our own infrastructure?
Yes — self-host any worker (a local process, your own server, a container you run) and nothing has to touch Query Farm’s infrastructure at all. Hosting through Orchard is a convenience, not a requirement.
Can we restrict who sees which rows or columns?
Yes — as a managed service we offer directly, not a DIY config: policy-driven, enforced adversarially, down to a single column. See “Row- and column-level security” below, and get in touch.
What if the source we need doesn’t have a worker yet?
Build one — SDKs exist for Python, TypeScript, Go, Rust, and Java, and the pattern is the same regardless of language — or ask us to build it for you.
Keep going