Shared storage backends

This page configures where shared state is stored. Your functions read and write that state through the BoundStorage handle (params.storage) — see Persist state across workers for what it’s for and the storage primitives it exposes. Here we set up the backend behind that handle, chosen with the VGI_WORKER_SHARED_STORAGE environment variable.

Backends

Backend	Value	Use Case	Dependencies
SQLite	`sqlite` (default)	Local / subprocess transport	None (stdlib)
Azure SQL	`azure-sql`	Azure cloud deployments	`vgi-python[azure]`
Cloudflare Durable Objects	`cloudflare-do`	Edge / multi-cloud deployments	None (stdlib)

SQLite (default)

Used automatically for subprocess transport. All workers share a local SQLite database file via WAL mode.

# No configuration needed — this is the default
vgi-serve my_worker.py

The database is stored at the platform-specific state directory (~/.local/state/vgi/vgi_storage.db on Linux). No setup required.

Azure SQL Database

For Azure cloud deployments (App Service, Container Apps, AKS) where workers run on separate hosts.

Setup

Install the Azure extra:

pip install "vgi-python[azure]"

Create an Azure SQL Database (Serverless recommended for cost):

az sql server create --name myserver --resource-group myrg --location eastus2 \
  --admin-user vgiadmin --admin-password 'MyPassword!'
az sql db create --name vgi --server myserver --resource-group myrg \
  --edition GeneralPurpose --compute-model Serverless \
  --family Gen5 --capacity 1 --auto-pause-delay 60 --min-capacity 0.5

Create the storage tables (once, during deployment):

from vgi.function_storage_azure_sql import FunctionStorageAzureSql

storage = FunctionStorageAzureSql(
  server="myserver.database.windows.net",
  database="vgi",
  user="vgiadmin",
  password="MyPassword!",
)
storage.ensure_tables()

Configure the worker via environment variables:

VGI_WORKER_SHARED_STORAGE=azure-sql
VGI_AZURE_SQL_SERVER=myserver.database.windows.net
VGI_AZURE_SQL_DATABASE=vgi
VGI_AZURE_SQL_USER=vgiadmin
VGI_AZURE_SQL_PASSWORD=MyPassword!

For managed identity (no username/password), omit VGI_AZURE_SQL_USER and VGI_AZURE_SQL_PASSWORD. The client will use DefaultAzureCredential.

Environment variables

Variable	Description
`VGI_AZURE_SQL_SERVER`	Server hostname (required)
`VGI_AZURE_SQL_DATABASE`	Database name (required)
`VGI_AZURE_SQL_USER`	SQL auth username (omit for managed identity)
`VGI_AZURE_SQL_PASSWORD`	SQL auth password (omit for managed identity)
`VGI_AZURE_SQL_DEBUG_LOG`	File path for debug/timing logs

Programmatic usage

from vgi.function_storage_azure_sql import FunctionStorageAzureSql

storage = FunctionStorageAzureSql(
  server="myserver.database.windows.net",
  database="vgi",
  user="vgiadmin",
  password="MyPassword!",
)

class MyTableFunction(TableFunctionGenerator):
  storage = storage

Cloudflare Durable Objects

For edge deployments and multi-cloud setups. Uses a Cloudflare Worker + Durable Object running SQLite internally. The DO is single-threaded, so all operations are inherently atomic without locking.

Setup

Deploy the Cloudflare Worker. The Worker source lives in a separate repository: vgi-cloudflare-durable-object-storage.

git clone https://github.com/query-farm/vgi-cloudflare-durable-object-storage
cd vgi-cloudflare-durable-object-storage
npm install
npx wrangler deploy

Set a bearer token for authentication:

npx wrangler secret put VGI_STORAGE_TOKEN

Configure the worker via environment variables:

VGI_WORKER_SHARED_STORAGE=cloudflare-do
VGI_CF_DO_URL=https://vgi-storage.myaccount.workers.dev
VGI_CF_DO_TOKEN=my-secret-token

No table creation step needed — the Durable Object creates its SQLite tables automatically on first request.

Environment variables

Variable	Description
`VGI_CF_DO_URL`	Cloudflare Worker URL (required)
`VGI_CF_DO_TOKEN`	Bearer token for authentication (optional)
`VGI_CF_DO_DEBUG_LOG`	File path for debug/timing logs

Programmatic usage

from vgi.function_storage_cf_do import FunctionStorageCfDo

storage = FunctionStorageCfDo(
  url="https://vgi-storage.myaccount.workers.dev",
  token="my-secret-token",
)

class MyTableFunction(TableFunctionGenerator):
  storage = storage

How it works

A single Durable Object instance handles all executions. Since execution_id is UUID4 (globally unique), there are no collisions between concurrent executions sharing the same DO. The DO uses the same SQLite schema as the local FunctionStorageSqlite backend.

Cleanup is handled by an hourly alarm that removes entries older than 24 hours.

Performance

The Cloudflare DO backend adds one HTTP round-trip per storage operation. Latency depends on proximity to the nearest Cloudflare PoP:

Location	Approx. per-operation latency
Co-located (same region)	5-15ms
Same continent	30-60ms
Cross-continent	80-150ms

Custom backends

A backend implements the FunctionStorage protocol: a key–value store, an append-log, atomic counters, and a work queue — the same primitives the BoundStorage handle exposes to your functions, all keyed by execution_id. The methods, grouped by category:

from vgi.function_storage import FunctionStorage

class MyCustomStorage:
  # key-value store
  def state_put_many(self, ...): ...
  def state_get_many(self, ...): ...
  def state_scan(self, ...): ...
  def state_drain(self, ...): ...
  def state_delete(self, ...): ...

  # append-log
  def state_append(self, ...): ...
  def state_log_scan(self, ...): ...

  # atomic counters
  def state_counter_add(self, ...): ...
  def state_counter_get(self, ...): ...
  def state_counter_set(self, ...): ...
  def state_counter_delete(self, ...): ...

  # work queue
  def queue_push(self, ...): ...
  def queue_pop(self, ...): ...
  def queue_clear(self, ...): ...

  # lifecycle
  def execution_clear(self, ...): ...

See the vgi.function_storage API reference for the exact signatures. Assign your backend to your function classes:

class MyFunction(TableFunctionGenerator):
  storage = MyCustomStorage()

Next steps

What the storage is for → Persist state across workers.
The functions that need it → Function patterns → Buffering.
Exact methods → API Reference: function_storage.