Running a worker

Java is the one SDK where how you run it needs as much attention as what you write. A JVM has a startup cost, needs flags Arrow depends on, and can hand batches through shared memory when the JDK is new enough.

The build settings that are not optional

tasks.withType<JavaCompile> {
  options.compilerArgs.add("-parameters")
}

application {
  applicationDefaultJvmArgs = listOf(
      "--add-opens=java.base/java.nio=ALL-UNNAMED",
      "--enable-native-access=ALL-UNNAMED",
  )
}

Setting	Why	Symptom if missing
`-parameters`	The annotation API reads parameter names to build the SQL signature.	Silent. Arguments are named `arg0`; positional calls still work, named calls stop resolving.
`–add-opens=java.base/java.nio=ALL-UNNAMED`	Arrow reaches into `java.nio` internals.	An `IllegalAccessError` at first use.
`–enable-native-access=ALL-UNNAMED`	The shared-memory transport makes native calls.	A native-access warning, then a fall back to the pipe.

Only the first fails quietly, which is why it is worth setting before writing a single function. See the tutorial for what it looks like when it is missing.

Use `launch:`

A cold JVM takes seconds to start. A bare LOCATION spawns a fresh one for every query:

-- A new JVM per query. Unusable interactively.
ATTACH 'demo' (TYPE vgi, LOCATION '/abs/path/bin/demo');

-- One JVM, reused across queries over a flock-coordinated Unix socket.
ATTACH 'demo' (TYPE vgi, LOCATION 'launch:/abs/path/bin/demo');

A pooled worker outlives your rebuild

The flip side of reuse: after ./gradlew installDist, the running worker is still the old build. It keeps answering until it idles out, so a fix can appear not to work for reasons that have nothing to do with the fix.

DETACH, kill the JVM, then re-attach. This is worth internalising early — it is the single most confusing thing about developing a Java worker, and it looks exactly like a bug in your code.

The four transports

Worker.builder()…runFromArgs(args) reads argv and picks one, so a single binary covers all of them:

Arguments	Transport	When
(none)	stdin/stdout	The default. The engine spawns and owns the process.
`–unix PATH`	AF_UNIX socket	What `launch:` uses. A warm worker, reused.
`–http PORT`	HTTP	Remote, shared, or load-balanced.
`–idle-timeout SEC`	modifier	How long a socket worker waits before exiting. `0` keeps it up.

Shared memory, on JDK 22+

By default Arrow batches travel over the transport’s pipe, and for large batches that copy dominates. VGI can instead pass them through a POSIX shared-memory segment both processes map.

It needs no code changes in your worker — it is negotiated at the transport layer. Set one environment variable on the engine side:

export VGI_RPC_SHM_SIZE_BYTES=67108864   # 64 MiB

The client creates and owns the segment and advertises it on each init; the worker attaches it, writes large batches into it, and sends a zero-row pointer batch down the pipe carrying an offset and length. The same path runs inbound for large input batches.

It degrades rather than fails

If a batch does not fit, is dictionary-encoded, or the segment is full, the transport falls back to the inline pipe. Correctness never depends on shared memory being available — which also means a misconfiguration costs you throughput silently rather than breaking a query. Check --enable-native-access is set if you expected the fast path and did not get it.

Next steps

What a worker exposes → Function patterns.
Exact signatures → Worker & serving.