Batch helpers

On this page

Building, reading, projecting and serializing record batches.

function adoptArrowJsShape

source

export function adoptArrowJsShape<T>(batch: T): T

Description

No-op on this backend: arrow-js RecordBatch/Vector already expose the full API worker code uses, and arrow-js is a peer dependency, so there is only ever one copy. Present so both backends satisfy one facade interface — see impl-flechette/compat.ts for what this does there.

const backend

source

const backend: VgiBackendInfo = { name: “arrow-js” }

function batchFromColumns

source

export function batchFromColumns(
columns: Record<string, any[]>,
schema: Schema | VgiSchema,
repr: "rich" | "raw" = "rich",
): RecordBatch

Description

Build a RecordBatch from column arrays (values are RICH). Each column is converted rich -> canonical via the codec, then canonical -> arrow-js column data via the canonical writer.

function batchFromRows

source

export function batchFromRows(
rows: Record<string, any>[],
schema: Schema | VgiSchema,
): RecordBatch

Description

Build a RecordBatch from row objects (values are RICH).

function batchToScalarDict

source

export function batchToScalarDict(
batch: RecordBatch | VgiBatch | null
): Record<string, any>

Description

Extract single-row batch to a scalar dict, in the RICH representation. Routes through the canonical reader + codec (same as iterRows) so a temporal/decimal setting is represented identically to column data and across backends, and so Dictionary-encoded columns (DuckDB sends these for enum-shaped fields like SchemaObjectType) are decoded — readCanonicalValue handles dictionary decode.

function batchToSecretDict

source

export function batchToSecretDict(
batch: RecordBatch | VgiBatch | null
): Record<string, Record<string, any>>

Description

Extract single-row batch to a secret dict (column per secret, each value is a struct). Handles both named secrets (column name = secret type) and scoped secrets (column name = “secret_N” with secret_type in field metadata).

function columnFromArray

source

export function columnFromArray(values: any[], type: VgiDataType): VgiColumnData

Description

Build an opaque column-data handle from a JS array. arrow-js wraps vectorFromArray and exposes its first Data node. Unlike the public build path, this is a low-level pass-through used by callers that already hold backend-native values, so it does NOT run the codec.

interface ColumnStatistics

source

export interface ColumnStatistics

Fields

columnNamestring

arrowTypeVgiDataType

minany

maxany

hasNullboolean

hasNotNullboolean

distinctCountbigint | number | null

containsUnicodeboolean | null

maxStringLengthbigint | number | null

function decodeDictValue

source

export function decodeDictValue(value: any, index = 0): any

Description

If value looks like a Dictionary-encoded Arrow scalar (Vector.get on a dict column on the apache-arrow fork returns the underlying Data, not the decoded string), pull out the decoded value at row index. Returns value unchanged when it isn’t dict-shaped.

Used at handler call sites where the incoming params came from the RPC layer’s row extractor (which doesn’t auto-decode dictionaries) and the handler needs the plain string.

function deserializeBatch

source

export function deserializeBatch(bytes: Uint8Array): RecordBatch

Description

Deserialize a RecordBatch from Arrow IPC bytes.

function deserializeSchema

source

export function deserializeSchema(bytes: Uint8Array): Schema

Description

Deserialize an Arrow Schema from IPC bytes. Note: In Bun, reader.schema is always undefined, so we must read from the batch.

Returns arrow-js’s Schema (which structurally satisfies VgiSchema).

function emptyBatch

source

export function emptyBatch(schema: Schema | VgiSchema): RecordBatch

Description

Create an empty (0-row) batch with the given schema. Accepts arrow-js Schema or facade VgiSchema (cast at the boundary).

function filterBatch

source

export function filterBatch(
batch: RecordBatch | VgiBatch,
mask: Uint8Array,
): RecordBatch

Description

Filter a RecordBatch using a Uint8Array mask (0=exclude, nonzero=include). Returns a new batch containing only the rows where mask[i] is nonzero.

Rows are read in canonical form then mapped back to RICH so the rebuild goes through the same codec/canonical path as every other column build — lossless and identical across backends (never the lossy/raw Vector.get).

function iterRows

source

export function* iterRows(
batch: RecordBatch | VgiBatch,
repr: "rich" | "raw" = "rich",
): Generator<Record<string, any>>

Description

Iterate rows of a RecordBatch as plain objects, in the RICH representation (Date for date32/date64; canonical otherwise). Reads via the per-backend canonical reader (lossless, backend-agnostic) then maps canonical -> rich through the codec — symmetric with the build path. Accepts arrow-js RecordBatch or facade VgiBatch.

function projectBatch

source

export function projectBatch(
projectionIds: number[] | null,
batch: RecordBatch | VgiBatch,
): RecordBatch

Description

Project a RecordBatch by column indices.

function projectSchema

source

export function projectSchema(
projectionIds: number[] | null,
schema: Schema | VgiSchema,
): Schema

Description

Project a schema by column indices, preserving only selected fields.

function readCanonicalValue

source

export function readCanonicalValue(
type: VgiDataType,
column: unknown,
index: number,
): unknown

Description

Read a single CANONICAL value at index from an arrow-js column (Vector). Reads the underlying typed-array storage for scalars (so it is lossless and never depends on Vector.get()’s lossy/divergent coercions — e.g. arrow-js Vector.get() returns a JS number for timestamp[us], losing precision), and recurses for composites.

function safeNumber

source

export function safeNumber(value: any): number

Description

Narrow a bigint to a number, refusing to do it lossily.

Arrow int64/uint64 values arrive as bigint. Several bind-time argument paths want a plain number, and used to get one from a bare Number(value) — which silently rounds above 2^53, so an id or a nanosecond timestamp passed as a function argument came back subtly wrong with nothing logged anywhere.

Throwing is the lesser evil: a caller that genuinely wants a lossy narrowing can still write Number(v) themselves, but nobody has to discover the loss from mismatched output weeks later.

function serializeBatch

source

export function serializeBatch(batch: RecordBatch | VgiBatch): Uint8Array

Description

Serialize a RecordBatch to Arrow IPC bytes. Accepts arrow-js RecordBatch or facade VgiBatch.

function serializeColumnStatistics

source

export function serializeColumnStatistics(
stats: ColumnStatistics[],
_cacheMaxAgeSeconds?: number | null,
): Uint8Array

Description

Serialize column statistics to IPC bytes per vgi-python’s wire format. Returns an empty-stats batch when stats is empty (matching Python).

function serializeSchema

source

export function serializeSchema(schema: Schema | VgiSchema): Uint8Array

Description

Serialize a Schema to Arrow IPC bytes. Accepts arrow-js Schema or facade VgiSchema (the latter is satisfied structurally by arrow-js Schema instances at runtime).

function toUint8Array

source

export function toUint8Array(val: any): Uint8Array

Description

Convert any binary-ish value to a Uint8Array. Returns empty array for null/undefined.