ADR 0001: Lance for vector-bearing and multimodal assets; embeddings are a column, not a kind¶

Status: Proposed
Date: 2026-06-30
Deciders: sunstone-py maintainers (@stigsb @Kesara03)

Context¶

sunstone-py models data as an Asset — a uniform envelope (payload, kind, metadata, extras) over a closed AssetKind enum (TABULAR, RASTER, ARRAY, TILES, BLOB, GEOFEATURES). The enum is deliberately closed: "New kinds (point clouds, meshes, audio) require adding a variant here. Plugin authors cannot extend this enum." Storage is fully decoupled behind two plugin protocols — FormatHandler (single-file: Parquet, .npz, GeoJSON…) and StoreFormatHandler (directory/location: Zarr, HDF5…) — discovered via a PluginRegistry over entry points, with built-ins loaded last so externals can override. Formats embed the sunstone Metadata (JSON-LD: slug, RDF prefixes, per-component schema, PROV-O lineage) in-file where the format allows.

We increasingly need to store embeddings and, more broadly, multimodal data: a raw document or image together with its extracted text, its vector representation, and structured metadata. Today vectors have no good home:

AssetKind.ARRAY is dict[str, ndarray] for scientific multidimensional data (climate grids, neuroimaging volumes) — free-floating named arrays with no notion of "row = an entity," and no similarity-search operation.
AssetKind.BLOB is opaque bytes — no schema, no index.

Neither supports the two things an embedding store actually needs: row-to-entity alignment (each vector travels with an id and its metadata/source) and ANN vector search (which needs an index).

Lance (Apache 2.0, Arrow-native, independent lance-format governance; file format 2.1 stable, pylance 7.x) is built for exactly this. It is a multimodal columnar format: scalars, native vector columns (FixedSizeList<float>), and large binary blobs co-exist as columns in one versioned dataset with fast random/range access, and it carries on-disk ANN indexes (IVF-PQ, IVF-HNSW) that live with the data on object storage. Parquet↔Lance conversion is a two-line, lossless Arrow round-trip. Caveats: optimistic/single-writer concurrency, compaction must be scheduled, the large-blob "Blob v2" encoding rides format 2.2 (still maturing), pylance is PyPI-labelled alpha despite production use, and its engine/tool ecosystem is narrower than Parquet's.

A prior decision must be reconciled, not contradicted: research-stack ADR 0003 already stores concept embeddings as an Iceberg list<float> table in data-platform's embeddings namespace, discovered by (scheme, model) and loaded into the extraction runtime for in-process similarity. This ADR must coexist with that.

Decision¶

1. Embeddings are a column type + index descriptor — not a new `AssetKind`¶

We do not add AssetKind.EMBEDDING. The closed enum stays as-is. A vector is a FixedSizeList<float> column on a TABULAR (or multimodal) asset, carrying an index descriptor — dimension, distance metric, index type and parameters — in the column's component_metadata (the same per-component channel already used for tabular columns, raster bands, and array variables).

Rationale: an embedding store is row/entity-aligned (row = chunk/document/image + id + metadata + optionally the source blob), which is TABULAR-shaped, not ARRAY-shaped. "Fixed-size float" is incidental; the load-bearing differences are the per-row identity and the search operation. "Searchable vector" is therefore a property of a column, not of a payload kind — which is also exactly how Lance models it. Keeping it column-level avoids a core enum change per use case and keeps the kind taxonomy about payload shape.

2. Lance is the default storage handler for vector-bearing and multimodal assets¶

Implement a LanceStoreHandler (directory-based StoreFormatHandler, __sunstone_handler_protocol__ = 2) supporting TABULAR and BLOB (and ARRAY where on-disk vectors/versioning are explicitly wanted). It:

embeds the sunstone Metadata JSON-LD in Lance schema metadata (same trick as ParquetFormatHandler);
persists vector columns and builds/loads the ANN index inside the .lance dataset;
registers via entry point, so it is discovered with no change to the registry.

3. Default handler is chosen per `(kind, context)`; Lance only where it is differentiated¶

"Preferred handler" means an overridable default (callers can still name a format), and the default depends on context — sunstone-py local files vs the data-platform Iceberg catalog:

Kind	sunstone-py default	data-platform default	Lance?
TABULAR	Parquet	Iceberg	only when vector-bearing
BLOB (incl. multimodal)	Lance or raw bytes	Lance / object-store sidecar	yes
ARRAY (sci. multidim)	Zarr / HDF5	Zarr sidecar	no (unless vectors+versioning)
RASTER	COG / GeoTIFF	COG sidecar	no
TILES	MBTiles / PMTiles	sidecar	no
GEOFEATURES	GeoParquet	GeoParquet sidecar	no

"Embeddings" is not a row in this table — it is a TABULAR/multimodal asset with a vector column (Decision 1), and that is the case where Lance becomes the default.

4. Lance does not replace Iceberg for plain tabular data¶

In data-platform, Iceberg stays the tabular spine (catalog identity, DuckDB attach, time-travel, schema evolution, the pointer-table design). Lance is a sidecar for vector/multimodal assets, not a second table format for ordinary tables — running both as competing table formats is operational tax with no incremental gain, and Lance's per-write versioning merely overlaps Iceberg snapshots.

5. Lance does not supersede the concept-embeddings catalog (research-stack ADR 0003)¶

The boundary is scale and access pattern, not dogma:

Stay Iceberg list<float> (ADR 0003): small, fixed (scheme, model) exact-lookup tables loaded into the extractor and searched in-process (the concept embeddings — thousands of rows).
Use Lance: large-scale, multimodal, or on-disk-ANN corpora — document / passage embeddings co-located with their source blobs, too big to hold in memory or wanting search to run against object storage.

The two coexist. If the concept-embeddings table ever outgrows in-memory search, migrating it to Lance is a two-line Arrow conversion (non-breaking) — at which point ADR 0003 is revisited, not before.

6. Lance versions are physical storage history, not authoritative provenance¶

Lance creates a new version per write; we treat that as storage-layer history for time-travel/debugging only. The PROV-O / JSON-LD lineage in Metadata remains the source of truth (consistent with data-platform ADR 0002 making the knowledge graph canonical). Do not build the provenance model on Lance versions.

7. Geospatial and scientific-grid kinds stay non-Lance¶

RASTER → COG/GeoTIFF, TILES → MBTiles/PMTiles, GEOFEATURES → GeoParquet, ARRAY → Zarr/HDF5. Lance has no geometry-native typing and would degrade these to WKB or tensor blobs, discarding window reads, overviews, reprojection, and per-tile access. These kinds keep their format-aware handlers.

Alternatives considered¶

Add AssetKind.EMBEDDING as a "stricter ARRAY." Rejected. It mistakes a column-level concern (a searchable vector) for a payload-shape concern, forces a core enum change, and fragments a multimodal row (blob + text + vector + metadata) into separate assets when Lance wants them as one row. An embedding table is TABULAR-shaped, not ARRAY-shaped.
Store embeddings as ARRAY (dict[str, ndarray]). Rejected. No row/entity alignment, nowhere natural for ids/metadata/index, no ANN. Technically expressible ({"vectors": (N,D), "ids": (N,)}) but it throws away the co-location and the index that make embeddings useful.
Lance as a general Parquet/Iceberg replacement for tabular. Rejected (Decision 4): two table formats = operational tax; versioning overlaps Iceberg; no incremental value for plain tables; Parquet/Iceberg remain the safer interchange/catalog substrate.
Adopt LanceDB (the embedded vector DB) rather than the Lance format. Deferred. The platform already owns a catalog and a lineage layer, so we take the format primitive (pylance) as a storage handler now and can layer LanceDB for serving later without changing the underlying files.

Consequences¶

New work in sunstone-py: a LanceStoreHandler; a vector-column type + index descriptor convention in component_metadata; and a vector-search (knn) query seam. Pin pylance versions given the alpha label and test upgrades.
data-platform auto-adopts via ContentRegistry.register_from_sunstone() with no code change — Lance assets ride the planned sidecar/pointer-table path as SIDECAR; the drift tests flag the new content descriptor. This fills the storage half of data-platform ADR 0002's "vector-search API seam" follow-up.
New capability: the multimodal document-corpus asset — one Lance dataset = raw document blob + text chunks + embeddings + claim/entity metadata, versioned together and ANN-searchable — becomes directly expressible.
Operational: compaction and old-version cleanup must be scheduled for Lance datasets; writes must be batched (single-writer/optimistic concurrency); validate ANN recall (notably IVF-HNSW defaults) on real embeddings before trusting defaults.
Maturity watch: the large-blob "Blob v2" encoding rides format 2.2 — validate large-media range reads before committing heavy media to Lance; Parquet/GeoParquet remain the choice for long-term interchange where any tool must read it.
Forward-compatible with ADR 0003: the concept-embeddings table can migrate Iceberg→Lance later as a lossless two-line Arrow conversion if it ever needs on-disk ANN.

Relationship to other ADRs¶

research-stack ADR 0003 (concept-embeddings catalog contract) — coexists; this ADR draws the Iceberg-vs-Lance boundary (Decision 5), does not supersede it.
data-platform ADR 0002 (graph is the canonical metadata store; vector-search API seam as a follow-up) — this ADR supplies the storage half of that seam and defers to the graph for canonical provenance (Decision 6).
data-platform ADR 0001 (sunstone URL scheme) — Lance datasets are addressed through the same object-store URI scheme.