Skip to main content

Overview

From: v10Status: Work-In-Progress

Apache Arrow utilities for luma.gl.

GitHub

Arrow Rendering​

These live examples exercise luma.gl's Arrow path, polygon, and text conversion and rendering stack through temporary deck.gl integration layers. They demonstrate progressive RecordBatch streaming, direct columnar inputs, constant and column styling, attribute rendering, and WebGPU storage rendering.

GitHub

API Reference​

Apache Arrow Preliminaries​

Apache Arrow has a rich type system that can represent a wide variety of binary data columns. A subset of these column types can be used directly as GPU vertex attribute data, meaning that such arrow columns can be uploaded efficiently to the GPU.

Apache Arrow supports primitive types like Float32, Uint32, and Uint8 that describe the value stored in each row. It also supports fixed-length vectors of these types with FixedSizeList. These scalar and fixed-length vector types map directly to the memory layouts used by GPU vertex attributes.

Arrow also supports variable-length List columns. These are useful for data such as polygons and paths, but they do not map directly to a single vertex attribute without an additional conversion step. PathAttributeModel provides the attribute-backed conversion for prepared Float32 XY, XYZ, and XYZM path coordinate rows, expanding each logical path into segment instances while keeping row-level style columns at the path boundary. PathStorageModel provides the WebGPU storage-backed form: compute expands GPU-resident path values into compact indexed segment records from copied list-offset metadata, render shaders fetch coordinates from the original path-value storage buffer, and per-path style rows remain storage bindings instead of being repeated for every generated segment. Use ArrowPathRenderer.convertToGPUVectors() or convertArrowPathToGPUVectors() to turn raw Float32 or Float64 Arrow path vectors into prepared path-attribute inputs. Use ArrowPathRenderer.convertToGPUVectors(..., {model: 'storage'}) or convertArrowPathStorageToGPUVectors() when WebGPU storage rendering should convert Float64 path payloads into Float32 deltas on the GPU before rendering.

GPU Table Interop​

@luma.gl/arrow is the adapter layer from Apache Arrow objects into the generic types in @luma.gl/tables:

  • makeGPUDataFromArrowData() uploads one Arrow Data chunk into a GPUData.
  • makeGPUVectorFromArrow() uploads one Arrow Vector into a GPUVector.
  • makeGPURecordBatchFromArrowRecordBatch() uploads one Arrow RecordBatch.
  • makeGPUTableFromArrowTable() uploads one Arrow Table while preserving source record batch boundaries.
  • makeGPUSplatDataFromArrow() prepares independently owned Gaussian splat batches for @luma.gl/splats, decoding GraphDECO encodings without clamping or quantizing spherical-harmonic DC radiance.
  • makeGPUSplatDataFromArrowStream() progressively prepares Gaussian splat batches without concatenating Arrow tables or rebuilding previously uploaded GPU buffers. Both Gaussian conversion helpers accept structurally compatible Arrow objects from other installed Arrow versions, including those produced by loaders.gl 5 alpha.
  • ArrowInputSchema combines source resolution and conversion with final GPUInputSchema validation for model-specific prepared inputs.

The resulting table schema is GPUSchema, and each vector has a GPUVector.format memory-layout string such as float32x3, unorm8x4, or vertex-list<float32x3>. Arrow DataType metadata may still be retained by adapter/readback paths, but @luma.gl/tables itself does not depend on apache-arrow.

See luma.gl for documentation.