GPUSegmentedLayout

Overview

GPUSegmentedLayout converts three packed binary flag streams into the offsets and counts needed to consume a nullable, segmented sequence without CPU readback.

At a glance

Question	Answer
Problem	Turn slot-aligned value, element, and segment-start flags into dense columnar layout metadata.
Reads / writes	Reads three packed binary flag streams; writes value and element offsets, segment indices and offsets, and three counts.
Ownership	All public inputs and outputs are caller-owned; hierarchical scan scratch is graph-owned transient memory.
Output contract	Exact source-aligned offsets plus a segment-offset prefix named by segmentCount.
Expected work	Three hierarchical scans, segment-offset publication over aligned spans, and one scalar-count pass.
Chunks	Six slot views may mix independent atomic/vector partitions; scans carry globally without packing. List offsets and counts remain atomic.
Conditions / budgets	Contributes ordinary graph nodes and does not compile, submit, read back, or publish results.
Neighborhood	format-specific classification → GPUSegmentedLayout → compaction, gather, nested layout, or rendering.

CostEvery slot is scanned three times even when few values are present or few segments are produced.

Common mistakeUse binary flags, keep segmentStartFlags[0] zero, and consume only segmentCount + 1 segment offsets.

When to use it

Use this operation after a parser, classifier, or application shader can answer three questions for every source slot:

• Does this slot own one physical value?
• Does this slot represent one logical element, including a null element?
• Does this slot start a new segment after the implicit first segment?

The result supports nullable-value compaction, list or group offsets, segmented follow-up work, and rendering that addresses an unpacked payload through dense indices. Apache Parquet definition and repetition levels are one source of these flags, but the operation has no Parquet-specific rules. Another columnar format can classify its own control data and reuse the same materialization stage.

Use GPUScan directly when only one prefix is needed. Use GPUCompaction when only a dense value or ID list is needed. GPUSegmentedLayout is useful when a downstream consumer needs value offsets, logical-element offsets, and group boundaries together.

Flag and output contract

All flag values must be exactly 0 or 1. A non-empty input has one implicit first segment, so segmentStartFlags[0] must be zero. Every later one starts a new segment. Empty input produces zero counts and writes segmentOffsets[0] = 0.

View	Length	Meaning
valueFlags	slot count	One when the slot owns a physical value
elementFlags	at least slot count	One when the slot represents a logical element
segmentStartFlags	at least slot count	One when the slot starts a segment after the first
valueOffsets	at least slot count	Exclusive dense physical-value index for each slot
elementOffsets	at least slot count	Exclusive dense logical-element index for each slot
segmentIndices	at least slot count	Inclusive scan of segment starts; the dense segment index
segmentOffsets	at least slot count + 1	Logical-element offset for every segment plus a terminal offset
valueCount	at least 1	Total physical values in element zero
elementCount	at least 1	Total logical elements in element zero
segmentCount	at least 1	Total segments in element zero

Only the first segmentCount + 1 entries of segmentOffsets are defined. The extra capacity lets the graph remain statically allocated even though the number of segments is data-dependent.

For example, these flags describe three segments, five logical elements, and four physical values:

valueFlags        [1, 0, 1, 1, 0, 1]
elementFlags      [1, 1, 1, 0, 1, 1]
segmentStartFlags [0, 0, 1, 0, 1, 0]

valueOffsets      [0, 1, 1, 2, 3, 3]
elementOffsets    [0, 1, 2, 3, 3, 4]
segmentIndices    [0, 0, 1, 1, 2, 2]
segmentOffsets    [0, 2, 3, 5]

Usage

import {GPUCommandGraph, GPUSegmentedLayout} from '@luma.gl/gpgpu/gpu-core';

const graph = new GPUCommandGraph(device, {id: 'column-layout'});

graph.add(new GPUSegmentedLayout({
  id: 'nullable-lists',
  valueFlags,
  elementFlags,
  segmentStartFlags,
  valueOffsets,
  elementOffsets,
  segmentIndices,
  segmentOffsets,
  valueCount,
  elementCount,
  segmentCount
}));

The operation contributes three GPUScan pipelines, segment-offset passes over aligned spans, and one count pass for the full sequence. The publication work is split so every shader stays within the eight-storage-buffer limit of the default WebGPU CORE profile. It does not compile the graph, submit commands, map counts, or move physical payload values.

Composition patterns

Pack present values

Feed valueFlags to GPUCompaction when the source is one packed uint32 value per slot. For wider records, use valueOffsets as stable scatter destinations in a format-specific kernel. The published valueCount names the valid output prefix.

Retain the original payload

If a later shader can address values in place, keep the payload unchanged and use valueOffsets to map logical slots to dense physical positions. This avoids a scatter and preserves zero-copy page buffers.

Build nested columns

Use segmentOffsets as one list-offset level. A format adapter can run another classification and layout operation for an ancestor depth, retaining one offset buffer per nesting level. Keep those buffers GPU-resident when the renderer or computation understands the nested representation.

Costs and limits

The operation performs three complete prefix scans plus one complete publication pass. It is most appropriate when several downstream stages reuse the resulting layout or when avoiding a CPU decode/readback/re-upload boundary matters more than a single lightweight CPU pass.

The six slot-aligned views may independently be packed GraphDataView<'uint32'> or GraphVectorView<'uint32'> values. Their chunk boundaries may differ. Scans carry across chunks, including segments spanning multiple chunks; empty chunks do not introduce segments. Alignment borrows views without concatenating or copying source storage.

valueFlags.length defines the active source range. Extra input capacity is ignored and extra slot-output capacity is left untouched. Re-encoding recomputes the layout and counts from current flags. Only the first logical row initializes the implicit segment, and only the final logical row publishes its terminal offset.

segmentOffsets and the three counts remain atomic destinations. Chunked final list-offset storage is follow-up work. Slot counts and offsets must fit in uint32; split larger datasets at existing page or batch boundaries.

GPUSegmentedLayout validates view formats, lengths, and graph ownership. It cannot cheaply inspect GPU-resident flag contents during graph construction, so the classifier is responsible for binary flags and the zero first-segment-start convention.