Skip to content
DuckDB Extensions All extensions ADBC Scanner

ADBC Scanner

Connect DuckDB to any database with an Arrow Database Connectivity (ADBC) driver — SQLite, PostgreSQL, Snowflake, Flight SQL, and more — using Arrow's columnar wire format end to end.

261,822
extension loads · last 90 days
On this page

Technical Overview

Arrow all the way from the remote database

Any ADBC-capable database becomes a table source in DuckDB, with Apache Arrow's columnar wire format end-to-end — no row conversions, no driver-specific glue.

What is ADBC?

Arrow Database Connectivity is a database-client API standard from the Apache Arrow project that moves data as Arrow columnar batches all the way from the database engine to your query. Where ODBC and JDBC convert rows on each side of the wire, ADBC keeps the columnar layout end-to-end.

  • • Standard API, many drivers: Drivers exist for SQLite, PostgreSQL, Snowflake, BigQuery, Flight SQL, DuckDB itself, and a growing list of others — anywhere data already lives in or near Arrow.
  • • No row-format intermediate: Result batches arrive as Arrow record batches and feed straight into DuckDB's vectorized executor without transposition or per-row allocation. A batch is a contiguous block of columns the driver hands DuckDB by pointer; DuckDB reads it in place.
  • • Pluggable driver model: Drivers are loaded as shared libraries (or via lightweight manifest.toml files) at runtime — no recompiling DuckDB to add support for a new database.

Two execution models

The same ADBC machinery is reachable two ways, and the choice is architectural, not cosmetic. Both can be loaded at once against the same database.

  • • Catalog mode — declarative, read-only: An ATTACH mounts the remote database as a DuckDB catalog: its schemas and tables appear under an alias, and the optimizer pushes projections and filters down into the driver while streaming Arrow batches into the vectorized executor. A join against local Parquet plans as one query — LIMIT can short-circuit the upstream scan, exactly like reading a Parquet file. The tradeoff is that the catalog interface is read-only — no DDL or DML flows through it.
  • • Function mode — imperative, full control: A connection handle lets you script the session explicitly: run server-side DDL/DML, set your own transaction boundaries, and perform Arrow-native bulk inserts that hand the driver columnar batches with no row-format detour. You trade automatic pushdown (here it only reaches as far as the SQL string you send) for the ability to write and to control transactions yourself.
  • • How to choose: Reach for catalog mode whenever the work is read-only analytics — it's less code and the optimizer does the pushdown for you. Drop into function mode the moment you need to mutate the remote, manage transactions explicitly, or bulk-load Arrow batches in. Because both target the same driver, a common pattern is catalog mode for the queries and function mode for the occasional write, in the same session.

Driver Setup

ADBC drivers ship as shared libraries (libadbc_driver_*.dylib / .so / .dll) plus an optional manifest.toml that names the symbol to load. The extension resolves them two ways depending on how you reference the driver.

  • • By name (manifest-resolved): 'driver': 'sqlite' looks up a .toml manifest in the standard ADBC search paths. Most install methods — pip install adbc-driver-postgresql, Homebrew, system packages — drop manifests in the right place automatically.
  • • By explicit path: 'driver': '/path/to/libadbc_driver_xxx.dylib' skips manifest resolution — useful for vendored drivers or unusual install locations.
  • • Custom search paths: Pass 'search_paths': '/opt/adbc/drivers' inside the connect options to add manifest directories without changing OS-level config.
  • • Drivers run as trusted native code: However it's resolved, a driver loads into DuckDB's address space and runs with the user's privileges — treat it like LOAD. In multi-tenant or hosted DuckDB, restrict ADBC access to roles that already have arbitrary-code-execution authority, and load only vetted upstream drivers.

ADBC Scanner vs Airport

Both extensions stream Arrow over the wire, but they speak different protocols and target different worlds. Use this section to pick the right tool — or load both, since they coexist cleanly.

  • • Use ADBC Scanner: When the system on the other end is a database with an ADBC driver — SQLite, PostgreSQL, Snowflake, BigQuery, MySQL, etc. ADBC is a client API standard.
  • • Use Airport: When the system on the other end speaks Arrow Flight directly — typically custom services or Flight-native systems like Dremio. Flight is an RPC protocol, not a client API. See the airport extension.
  • • vs purpose-built scanners: DuckDB's postgres_scanner / mysql_scanner / sqlite_scanner are often faster for the one database they target. Reach for ADBC Scanner when no purpose-built extension exists, when you want a single client surface across many backends, or when you specifically want Arrow end-to-end (Snowflake, BigQuery).

Deep Dive

Technical Details

Most users want the catalog interface. One ATTACH and the remote database appears as an ordinary DuckDB catalog — schemas, tables, projection and filter pushdown, all without writing any adbc_* function calls:

-- libpq reads credentials from the URI itself, not separate options.
ATTACH 'postgresql://reader:secret@localhost/analytics' AS pg (
  TYPE adbc,
  driver 'postgresql'
);


SELECT user_id, COUNT(*)
FROM pg.public.activity
WHERE ts >= CURRENT_DATE - 7
GROUP BY user_id;

The full parameter list and more examples live in the Catalogs section. Catalog mode is read-only — for DDL, DML, transactions, or bulk Arrow-native inserts, drop down to function mode below.

Function mode (when you need to write)

When you need to mutate the remote, run server-side SQL, or do Arrow-native bulk inserts, adbc_connect returns a BIGINT handle that points at a driver-managed connection. Stash it in a SQL variable so you can reuse it across statements:

SET VARIABLE conn = (SELECT adbc_connect({'driver': 'postgresql', 'uri': :pg_uri}));

The handle is opaque — don’t compute on it, just pass it back into the other adbc_* functions. Connections are cleaned up when the DuckDB process exits, but call adbc_disconnect explicitly when you’re done — it releases the driver’s connection-pool slot immediately rather than waiting for process teardown.

Performance

The extension passes Arrow record batches straight from the driver into DuckDB’s vectorized executor with no transposition. Compared to a row-format client (ODBC, JDBC, libpq), three things change:

  • No serialize/deserialize per row. Each batch is a contiguous block of columns the driver hands DuckDB by pointer; DuckDB reads it in place.
  • Streaming. Results start producing rows as the first batch arrives — LIMIT short-circuits the upstream query if the driver supports it, just like reading a Parquet file. Applies to both catalog mode (via the table-scan path) and adbc_scan.
  • Push-down depends on the driver. In catalog mode, DuckDB pushes projections and filter predicates into the driver automatically; the driver and the remote database decide what to do with them. In function mode, push-down only goes as far as the SQL string you hand to adbc_scan. Use EXPLAIN on the remote database (via adbc_execute) when shape matters.

For adbc_insert, the same applies in reverse: DuckDB hands the driver Arrow batches it produced from your SELECT, so a SELECT * FROM big_table doesn’t materialize as rows on the way out.

Securing credentials

Connection options are plain SQL values, so anything you pass inline is visible in query logs and the catalog. The recommended pattern is parameterizing through SQL variables seeded from the environment, so the literal credential never appears in any cached SQL plan:

-- Catalog mode — libpq pulls user/pass from the URI directly. The URI
-- itself is a string literal (no `:bind_var` interpolation in ATTACH),
-- so build it elsewhere if you need to compose it from secrets.
ATTACH 'postgresql://reader:secret@localhost/analytics' AS pg (
  TYPE adbc,
  driver 'postgresql'
);


-- Function mode
SET VARIABLE conn = (SELECT adbc_connect({
  'driver':   'postgresql',
  'uri':      'postgresql://localhost:5432/analytics',
  'username': 'reader',
  'password': :pg_password
}));

getenv runs at parse time, so the literal password never appears in the cached SQL plan.

ADBC drivers are arbitrary native code

Drivers loaded by name resolve through the ADBC manifest search path; drivers loaded by absolute path skip that. In either case, the driver runs in DuckDB’s address space with the user’s privileges. Only load drivers you trust. In a multi-tenant or hosted DuckDB environment, treat both ATTACH ... (TYPE adbc, ...) and adbc_connect the same way you’d treat LOAD — restrict them to roles that already have arbitrary-code-execution authority. See the ADBC driver list for vetted upstream drivers.

Compared to alternatives

  • DuckDB’s postgres_scanner / mysql_scanner / sqlite_scanner — purpose-built per-database extensions, often faster for the database they target. Reach for ADBC Scanner when no purpose-built extension exists, when you need a single client surface across many backends, or when you specifically want Arrow-end-to-end (e.g., for Snowflake or BigQuery).
  • airport — the same Arrow-end-to-end principle but over Arrow Flight instead of ADBC. Use it when the upstream is Flight-native; use this when the upstream is a database with an ADBC driver.
  • Python pyarrow.flight / adbc_driver_* + DataFrames — much heavier setup; you give up DuckDB’s vectorized execution against the streamed data and re-introduce row-format conversion at the DataFrame boundary.

Install

INSTALL adbc_scanner FROM community;
LOAD adbc_scanner;

Quick Start

Connect to SQLite via its ADBC driver

SET VARIABLE conn = (SELECT adbc_connect({'driver':'sqlite', 'uri':':memory:'}));

Run a query — result is a regular DuckDB table

SET VARIABLE conn = (SELECT adbc_connect({'driver':'sqlite', 'uri':':memory:'}));
SELECT * FROM adbc_scan(getvariable('conn')::BIGINT, 'SELECT * FROM my_table');

Run DDL / DML on the remote database

SET VARIABLE conn = (SELECT adbc_connect({'driver':'sqlite', 'uri':':memory:'}));
SELECT adbc_execute(getvariable('conn')::BIGINT, 'CREATE TABLE users(id INT, name TEXT)');

Disconnect when done

SET VARIABLE conn = (SELECT adbc_connect({'driver':'sqlite', 'uri':':memory:'}));
SELECT adbc_disconnect(getvariable('conn')::BIGINT);

Reference

Extension Contents

Quick reference to all available functions and settings organized by category.

Name Description
ADBC Catalog
adbc Attach an ADBC-driven database as a DuckDB catalog.
Connection

Open, close, and inspect ADBC connections. Every other function takes a BIGINT connection handle returned by adbc_connect.
adbc_clear_cache() Drop any cached driver / connection state.
adbc_connect() Open a connection to a remote database via an ADBC driver.
adbc_disconnect() Close a connection opened with adbc_connect and free its resources.
adbc_info() Return driver and server metadata for an open connection — vendor name, version strings, supported features.
Database Connection
adbc Stored credentials and connection details for ADBC drivers.
Mutation

Write — DDL/DML through adbc_execute, plus bulk Arrow-native loads via adbc_insert.
adbc_execute() Execute a non-SELECT statement (DDL or DML).
adbc_insert() Bulk-insert into a remote table — passes Arrow record batches directly through the driver's bulk-load path.
Query

Read data from the remote database — full SQL queries (adbc_scan) or whole-table streams (adbc_scan_table).
adbc_scan() Execute a SELECT and return its rows as a DuckDB table.
adbc_scan_table() Stream an entire remote table by name, without writing SQL — equivalent to adbc_scan(conn, 'SELECT * FROM <table>') but routed through ADBC's bulk-read API where supported.
Schema

Inspect the remote catalog — tables, columns, table types, full Arrow schemas — through the driver's metadata API.
adbc_columns() Describe the columns of a table — name, type, nullability, default — pulled from the driver's catalog API.
adbc_schema() Return the full Arrow schema for a query result without executing it.
adbc_table_types() List the table-type categories the driver/server distinguishes (e.g.
adbc_tables() List tables visible to the connection.
Transactions

Manual transaction control. Toggle autocommit, then commit or roll back explicitly.
adbc_commit() Commit the current transaction on the connection.
adbc_rollback() Roll back the current transaction.
adbc_set_autocommit() Toggle autocommit mode.

API Reference

Function Documentation

Database Storage

Storage Extensions

Catalog implementations that attach external storage as a DuckDB database.

adbc

Catalog ADBC Catalog
Description

Attach an ADBC-driven database as a DuckDB catalog. Once attached, DuckDB sees the remote schemas and tables as local objects — projection and filter pushdown happen automatically, and the executor pulls Arrow batches straight from the driver. Use this when you want a database-style interface; for ad-hoc reads / writes / arbitrary SQL, the adbc_scan, adbc_execute, and adbc_insert functions stay available alongside.

Parameters
Parameter Type Required Description
driver VARCHAR Required Driver identifier — a registered name (e.g. 'sqlite', 'postgresql'), an absolute path to a libadbc_driver_* shared library, or a manifest name resolved via the ADBC search paths.
entrypoint VARCHAR Optional Custom driver entrypoint function name. Only needed for drivers that don't follow the standard AdbcDriverInit symbol convention.
search_paths VARCHAR Optional Extra colon-separated directories to look in for ADBC manifest (*.toml) files, in addition to the platform defaults.
use_manifests VARCHAR Optional Default: true Set to 'false' to skip manifest resolution entirely (load the driver only by absolute path).
batch_size INTEGER Optional Hint for the number of rows per Arrow batch when scanning. Larger values reduce per-batch overhead at the cost of memory.
Examples
1

Attach a PostgreSQL database via the postgresql ADBC driver

-- libpq pulls credentials from the URI; ATTACH parameters don't interpolate bind variables.
ATTACH 'postgresql://reader:secret@localhost/mydb' AS pg (
  TYPE adbc,
  driver 'postgresql'
);

-- Query as if it were local — projection + filter pushdown happen automatically
SELECT user_id, COUNT(*) AS events
FROM pg.public.activity
WHERE ts >= CURRENT_DATE - 7
GROUP BY user_id;
2

Attach a SQLite database file via the sqlite ADBC driver

ATTACH '/data/app.db' AS local_app (
  TYPE adbc,
  driver 'sqlite'
);

SHOW TABLES FROM local_app.main;
3

Attach using an explicit driver-library path (skips manifest resolution)

ATTACH 'snowflake://account.snowflakecomputing.com' AS sf (
  TYPE adbc,
  driver '/opt/adbc/lib/libadbc_driver_snowflake.dylib',
  username :sf_user,
  password :sf_password,
  use_manifests 'false'
);
Back to Extension Contents

Security

Secrets

DuckDB secrets for storing the credentials and keys used by the adbc scanner extension.

adbc

adbc Secret Database Connection
Description

Stored credentials and connection details for ADBC drivers. Reference a secret by name in adbc_connect, or rely on SCOPE-based URI lookup so the right secret is selected automatically when its scope prefix matches the connection URI.

Parameters
Parameter driver Type VARCHAR Required Required Description ADBC driver name (e.g. 'sqlite', 'postgresql') or path to a shared library.
Parameter uri Type VARCHAR Required Optional Description Connection URI passed to the driver. Driver-specific format.
Parameter username Type VARCHAR Required Optional Description Database username.
Parameter password Type VARCHAR Required Optional Description Database password. Automatically redacted in logs.
Parameter database Type VARCHAR Required Optional Description Database name, when not encoded in the URI.
Parameter entrypoint Type VARCHAR Required Optional Description Custom driver entrypoint function name (rarely needed).
Examples
1

PostgreSQL with credentials and a SCOPE for auto-lookup

CREATE SECRET my_postgres (
  TYPE adbc,
  SCOPE 'postgresql://prod-server:5432',
  driver 'postgresql',
  uri 'postgresql://prod-server:5432/mydb',
  username 'app_user',
  password 'secret_password'
);
2

Local SQLite

CREATE SECRET my_sqlite (
  TYPE adbc,
  SCOPE 'sqlite://data',
  driver 'sqlite',
  uri '/var/data/app.db'
);
3

Use the secret explicitly by name

SET VARIABLE conn = (SELECT adbc_connect({'secret': 'my_postgres'}));
4

Auto-lookup — DuckDB picks the secret whose SCOPE matches the URI

SET VARIABLE conn = (SELECT adbc_connect({
  'uri': 'postgresql://prod-server:5432/mydb'
}));
5

Persist the secret to ~/.duckdb/secrets/

CREATE PERSISTENT SECRET my_postgres (
  TYPE adbc,
  SCOPE 'postgresql://prod-server:5432',
  driver 'postgresql',
  uri 'postgresql://prod-server:5432/mydb',
  username 'app_user',
  password 'secret_password'
);
Back to Extension Contents

Practical Examples

Cookbook

Real-world recipes and patterns for common use cases.

There are two ways to use the extension. Pick by what you’re doing:

  • Catalog mode (ATTACH ... TYPE adbc) — read-only, but the remote database appears as a normal DuckDB catalog with automatic projection and filter pushdown. Best for plain SELECT-style analytics.
  • Function mode (adbc_connect + adbc_scan / adbc_execute / adbc_insert) — full read/write access including DDL, DML, and bulk Arrow-native inserts. Best when you need to mutate the remote, run server-side SQL, or hold a long-lived connection.

Both can coexist in the same session.

Attach a remote database (catalog mode)

-- PostgreSQL via the postgresql ADBC driver. libpq reads its credentials
-- from the URI (postgresql://user:pass@host/db), so embed them there
-- rather than passing username/password as separate ATTACH parameters.
ATTACH 'postgresql://reader:secret@localhost/analytics' AS pg (
  TYPE adbc,
  driver 'postgresql'
);


-- Query as if it were a local DuckDB catalog
SELECT user_id, COUNT(*) AS events
FROM pg.public.activity
WHERE ts >= CURRENT_DATE - 7
GROUP BY user_id;
-- SQLite file via the sqlite driver
ATTACH '/data/app.db' AS local_app (TYPE adbc, driver 'sqlite');
SHOW TABLES FROM local_app.main;

Catalog mode is read-only — see Catalogs for the full parameter list. Reach for function mode below when you need to write.

Open a connection (function mode)

-- By driver name (driver manifest must be installed)
SET VARIABLE conn = (SELECT adbc_connect({
  'driver': 'sqlite',
  'uri':    ':memory:'
}));


-- Or by explicit driver path
SET VARIABLE conn = (SELECT adbc_connect({
  'driver': '/path/to/libadbc_driver_sqlite.dylib',
  'uri':    '/data/app.db'
}));


-- PostgreSQL with credentials
SET VARIABLE conn = (SELECT adbc_connect({
  'driver':   'postgresql',
  'uri':      'postgresql://localhost:5432/analytics',
  'username': 'reader',
  'password': :pg_password
}));

Stash the connection handle in a SQL variable and reuse it across statements.

Query

-- Run any SELECT
SELECT * FROM adbc_scan(getvariable('conn')::BIGINT,
                        'SELECT * FROM orders WHERE created_at > now() - INTERVAL 1 DAY');
-- Or stream a whole table by name
SELECT * FROM adbc_scan_table(getvariable('conn')::BIGINT, 'orders');

Result is a normal DuckDB table — JOIN against local tables, persist with CTAS, re-export.

Mutation

-- DDL
SELECT adbc_execute(getvariable('conn')::BIGINT,
                    'CREATE TABLE users(id INT, name TEXT)');


-- DML
SELECT adbc_execute(getvariable('conn')::BIGINT,
                    'INSERT INTO users VALUES (1, ''Alice'')');


-- Bulk insert from a DuckDB query — Arrow-native, no row-format conversion.
-- adbc_insert is a TABLE function, so it goes in the FROM clause.
SELECT * FROM adbc_insert(
  getvariable('conn')::BIGINT,
  'users',
  (SELECT id, name FROM staging_users)
);

Transactions

SELECT adbc_set_autocommit(getvariable('conn')::BIGINT, FALSE);


SELECT adbc_execute(getvariable('conn')::BIGINT, 'INSERT INTO orders VALUES (...)');
SELECT adbc_execute(getvariable('conn')::BIGINT, 'UPDATE inventory SET qty = qty - 1 WHERE sku = ''X''');


SELECT adbc_commit(getvariable('conn')::BIGINT);
-- or adbc_rollback on error

Inspect the remote schema

-- All tables
SELECT * FROM adbc_tables(getvariable('conn')::BIGINT);
-- Columns of a specific table (filters are named parameters)
SELECT * FROM adbc_columns(getvariable('conn')::BIGINT, table_name := 'orders');
-- Full Arrow schema of a query without running it
SELECT * FROM adbc_schema(getvariable('conn')::BIGINT, 'SELECT * FROM orders');

Useful for building dynamic dashboards or schema-driven ETL pipelines.

Disconnect

SELECT adbc_disconnect(getvariable('conn')::BIGINT);

Connections are also cleaned up when the DuckDB process exits, but explicit disconnect is good hygiene.

Platform Support

Compatibility

Extension availability may vary by platform and DuckDB version. Check below to ensure this extension supports your environment before installation.

Quick Facts

Release status Stable
Software License MIT
Pricing Free
Written In C++
Source Available Yes
View on GitHub
Usage
261,822
loads · last 90 days

Platforms

  • Linux x86_64 aarch64
  • Linux (musl) Not available
  • macOS Intel Apple Silicon
  • Windows x86_64
  • WASM Not available
Compiled binary sizes
Platform Architecture Size
Linux x86_64 11.14 MB
Linux aarch64 9.87 MB
macOS Intel 8.47 MB
macOS Apple Silicon 7.46 MB
Windows x86_64 7.64 MB

Compressed download size from the Haybarn extension repository.

DuckDB & Haybarn

Release calendar