Skip to content

Asset Envelope — Open Design Decisions

Date: 2026-05-12 Status: Draft — awaiting decisions Parent spec: 2026-05-12-generic-format-handler-asset-envelope-design.md

Each section is one decision that needs a call before the parent spec can ship. The items came out of a codex peer-review pass; the parent spec already applied the "just-fix" findings (factual errors, mechanical bugs). What remains here is genuinely load-bearing design — pick a direction or push back on the framing.

Leave inline comments on the decisions you want changed. Comments on a single decision take precedence over the recommendations below.

1. DataFrame wrapper vs. Asset ownership (tabular kind)

Problem. The current sunstone.DataFrame is a wrapper holding pd.DataFrame in .data and a Metadata in .metadata. Legacy FormatHandler.read() returns a bare pd.DataFrame. The parent spec needs to declare which object owns which for tabular assets, and how df.metadata stays in sync with asset.metadata.

Options.

  • A. Asset owns. sunstone.DataFrame becomes a thin facade over an Asset. df.data returns asset.as_table(); df.metadata returns asset.metadata. Legacy handlers continue returning pd.DataFrame; the adapter wraps. New handlers return Asset(kind=TABULAR, payload=pd.DataFrame).
  • B. sunstone.DataFrame owns. Asset is a transient handoff at handler boundaries. Inside the library, tabular data flows as sunstone.DataFrame. Asset only exists in ss.read/ss.write/derive() calls. Non-tabular kinds use Asset end-to-end.
  • C. Parallel containers with explicit sync points. Asset.metadata is df.metadata is enforced at Asset construction and after every derive(). Users mutate either side and it shows up everywhere. Requires a sync helper and rules for when to re-sync (e.g., after pandas operations that drop attrs).

Recommendation: A. Single source of truth eliminates "which one do I edit". The facade keeps the user-facing ergonomics (df.metadata.description = ...) intact. Cost: a moderate refactor of sunstone.DataFrame to delegate everywhere.

2. Mapping sugar on Metadata

Problem. Examples in the parent spec used asset.metadata["sosa:observedProperty"] = .... Metadata doesn't implement __getitem__/__setitem__. The current workaround is asset.metadata.custom_properties = asset.metadata.custom_properties or {} then key access — clunky.

Options.

  • A. Add __getitem__/__setitem__/__delitem__/__contains__ to Metadata that proxy to custom_properties (lazy-init the dict on first set). Treat colon-bearing keys as RDF triples; non-namespaced keys raise.
  • B. Leave Metadata as-is, document the verbose form. Users learn the API.
  • C. Add a separate asset.rdf[...] mapping that proxies to custom_properties, leaving Metadata itself unchanged.

Recommendation: A. This is the discovery story's primary touch-point — data scientists will write asset.metadata["sosa:observedProperty"] = ... constantly. The mapping sugar is a few lines and matches the way users already think about RDF triples on a dataset.

3. derive() — multi-parent

Problem. Many real operations combine assets: raster mosaics, image stacks, table joins, tile generation from multiple sources. derive() as drafted assumes a single implicit parent.

Options.

  • A. Asset.derive(payload, *, derived_from: Iterable["Asset"] | None = None, ...). When derived_from is None, default to [self]. When provided, record one prov:wasDerivedFrom per parent.
  • B. Class method Asset.combine(parents, payload, ...). self.derive(...) remains single-parent; explicit combinator for multi-parent.
  • C. Defer. Single-parent only in v1; revisit when first multi-parent handler ships.

Recommendation: A. The default stays clean (asset.derive(payload=...) still works), and lineage stays correct for mosaics/joins. PROV-O models multi-parent derivation natively; no reason to artificially restrict it.

4. derive() — name inheritance

Problem. Parent spec clears slug to None on derive but leaves name inherited. A derived NDVI asset would silently carry the parent's "Sentinel-2 SR, July 2024" name unless the user overrides.

Options.

  • A. Clear both slug and name to None by default. User must supply both on the derive() call or before write.
  • B. Inherit name by default. User overrides when they care.
  • C. Add inherit_identity: bool = False — explicit opt-in for inheritance.

Recommendation: A. Same logic as slug: a new dataset deserves its own name. Forcing the user to think about it at derive time matches the existing to_csv(name=...) contract.

5. derive() — custom_properties propagation

Problem. Blindly carrying custom_properties forward can make false statements. Example: parent has sosa:observedProperty = "surface-reflectance". NDVI derivative inherits it unless overridden — but NDVI is not surface reflectance.

Options.

  • A. Split into two buckets on Metadata. custom_properties (semantic, per-asset) and provenance_properties (carries forward through derive). RDF emission joins them on serialise.
  • B. Add an inheritance manifest. Each RDF key declares its inheritance behaviour ("inherit", "clear", "ask"). Driven by a small built-in table for known prefixes (sosa:, dcat:, prov:, ...) plus user overrides.
  • C. Don't inherit by default; require derive(..., inherit_custom_properties=True).
  • D. Inherit by default; document the footgun.

Recommendation: C. Semantic claims about data are not provenance. Forcing explicit opt-in is the safe default. Option A is cleaner long-term but adds API surface and storage complexity; do it when (B/C) proves insufficient.

6. Extras mutability on inheritance

Problem. extras is dict[str, Any]. derive() as drafted inherits it shallowly, so parent and child can share a mutable profile dict or arrays map. Mutating the child's profile silently mutates the parent's.

Options.

  • A. Deep-copy extras on derive. Safe default; small perf cost on big extras.
  • B. Shallow-copy + document the gotcha.
  • C. Make extras values immutable by convention (frozen dataclasses, tuples). Burdens plugin authors.

Recommendation: A. Deep-copy. The kinds we expect — profile dicts, CRS strings, chunk specs — are small. The bug class this prevents (silent cross-asset mutation) is nasty and hard to diagnose.

7. Per-kind derive policies (raster profile invalidation)

Problem. Inheriting extras["profile"] for raster derive() is unsafe when the payload shape/dtype/band count changes. NDVI from a multiband uint16 image becomes single-band float; the inherited profile lies about count, dtype, nodata.

Options.

  • A. Asset has no policy; writers validate and raise. Cheap to implement; user finds out at write time.
  • B. Kind-specific derive policies. AssetKind.RASTER derive inspects the payload shape vs. parent, drops stale profile keys (count, dtype, nodata), keeps geo keys (transform, crs).
  • C. Helper free functions per kind (raster_derive(asset, payload, ...)) that encapsulate the right invalidation. Asset.derive() stays naive.

Recommendation: B. A small registry of KindDerivePolicy callables keyed on AssetKind, called from Asset.derive(). Raster gets the auto-invalidation logic; other kinds get a no-op default until they need it.

8. derive() from unsaved parents

Problem. prov:wasDerivedFrom referencing a parent that was never written (no slug, no location) is not discoverable or serialisable as a useful entity.

Options.

  • A. Reject. Raise on derive() if self.metadata.slug is None.
  • B. Use content-hash identity. If parent has no slug, record prov:wasDerivedFrom <_:hash-of-content> as a blank node.
  • C. Collapse to parent's sources. Skip the immediate parent; record what it was derived from. Loses one hop of lineage.
  • D. Warn and proceed. Record the parent as best you can (in-memory id) and let the user know the lineage will be incomplete.

Recommendation: B + D. Content-hash blank nodes preserve the lineage shape; emit a warning the first time it happens per process. Future work: opt-in "materialise upstream" mode that writes intermediate assets to disk to give them durable identity.

9. Array payload vs. extras["arrays"] overlap

Problem. Kind taxonomy says AssetKind.ARRAY payload is dict[str, ndarray], but asset.arrays is also defined as a convenience accessor over extras["arrays"]. Same data in two places.

Options.

  • A. Drop asset.arrays from D3. For ARRAY kind, users access asset.as_array() (returns the payload dict directly). extras reserved for store metadata (compressed, dimension labels, chunking).
  • B. Drop the payload form; store arrays only in extras. Payload becomes a store handle. Discovery code uses asset.arrays. Inconsistent with how raster works (payload IS the array there).
  • C. Keep both; make asset.arrays an alias for as_array().

Recommendation: A. Payload IS the data, for every kind. Extras hold structure/ metadata about the data. The duplication is a leftover from earlier drafts.

10. Tile pyramids and the stream-based handler protocol

Problem. FormatHandler.read(stream: BinaryIO) / write(asset, stream) doesn't fit XYZ tile directories, MBTiles (single SQLite file but pattern-based access), Zarr stores (directory of chunks), or object-store prefixes. Tiles need store/location access, not a single byte stream.

Options.

  • A. Defer tiles. Drop AssetKind.TILES from the v1 enum. Add it in a later spec that extends the handler protocol with a read_store(path) method.
  • B. Add a parallel store-based protocol now. StoreFormatHandler with read(location, **kw) -> Asset and write(asset, location, **kw) -> None. URLHandler provides "is this a directory/store?" classification.
  • C. Generalise stream to a Resource abstraction — Resource.open_byte_stream() for single-file formats, Resource.list() / Resource.subpath() for stores. Handlers declare which they need.

Recommendation: B. Keeping TILES in v1 the enum but with no shipping handler is fine; the protocol gap is what blocks shipping. A clean parallel protocol avoids forcing single-file formats through a more complex abstraction, and lets URLHandler extensions (GCS prefix listing, etc.) plug in cleanly. C is more elegant but premature until we have two store-based formats to compare.

11. Autodispatch beyond file extensions

Problem. ss.read("inputs/sentinel2_2024_07.tif") works via extension. But .zarr is a directory, XYZ tile pyramids have no canonical extension, cloud prefixes (gs://bucket/path/) often lack one. Extension dispatch breaks.

Options.

  • A. Multi-key dispatch. Handlers declare can_read(path, format, *, store_kind). Registry tries extension, then store kind (directory, file, URL-prefix), then declared format argument. First match wins.
  • B. Explicit kind= parameter on ss.read. When auto-detection fails, the user passes ss.read(path, kind=AssetKind.TILES).
  • C. Consult datasets.yaml first. If the dataset entry declares format: mbtiles, use that to pick the handler. Falls back to extension.

Recommendation: C, then A as fallback. datasets.yaml is already the source of truth for what a dataset is; honouring its declared format is the cleanest signal. Extension dispatch stays for ad-hoc reads outside of datasets.yaml.

12. Legacy vs. new handler distinction

Problem. The current parent spec proposed return-annotation inspection (if read() returns Asset → new; else → legacy). Unreliable — old plugins lack annotations, and a runtime call consumes a real stream.

Options.

  • A. Two protocols. FormatHandler (legacy, returns pd.DataFrame) and AssetFormatHandler (new, returns Asset). Registry tries both isinstance checks on plugin instantiation; never inspects return annotations or runtime returns.
  • B. Capability marker. New-style handlers set a class attribute __sunstone_format_protocol__ = 2. Registry checks the attribute; absence → legacy.
  • C. Separate entry-point group. sunstone.format_handlers (new) vs. sunstone.plugins (legacy). Migration = move the entry-point declaration.

Recommendation: A. Two protocols is the most Python-idiomatic, plays well with runtime_checkable, and gives plugin authors a clear "switch your base class" migration target. B is cute but ad-hoc; C creates a long-tail of plugins registered in the wrong group.

13. supports_metadata() — one capability or two?

Problem. Today's supports_metadata() conflates two things: "I can extract native file metadata (CRS, transform, band tags from a GeoTIFF)" and "I can embed Sunstone JSON-LD into the file". A GeoTIFF handler can do the first but not the second.

Options.

  • A. Split into two predicates. supports_native_metadata() and supports_sunstone_metadata(). Read paths consult the first, write paths consult the second.
  • B. One predicate stays; add a metadata_capabilities() set. Returns a set of capability flags. Forward-compatible if we add more later.
  • C. Keep one predicate, interpret it as Sunstone-only. Native metadata extraction is implicit in what the handler chooses to populate on the Asset.

Recommendation: A. Clear, two predicates, easy to migrate. B is over-engineered for the current need; C silently loses an important capability distinction.

14. PluginRegistry.get_format_handlers() migration risk

Problem. External code (not just plugin authors — users of the registry) calls get_format_handlers() and invokes handler.read() expecting pd.DataFrame. Switching the contract to Asset silently breaks them.

Options.

  • A. Add new accessors; deprecate the old. get_asset_format_handlers() returns new-protocol handlers. get_format_handlers() keeps returning legacy ones (or adapters thereof). Deprecation warning on the latter.
  • B. Break it; document loudly in CHANGELOG.
  • C. Provide a read_dataframe() / read_asset() helper pair on each handler that works for both flavours via the adapter.

Recommendation: A. Plugin API is a library promise. A versioned migration path (new accessor + deprecation) keeps trust intact.

15. RDF discovery shape (per-kind required properties)

Problem. The discovery destination is "find datasets by language". custom_properties: dict[str, Any] is too unstructured: nothing forces a RASTER asset to declare CRS, bbox, temporal coverage, bands. Without a required RDF shape per kind, discovery queries will either fail or hallucinate.

Options.

  • A. Per-kind RDF profiles, validated at write. Define minimum required triples per AssetKind:
  • TABULAR: dcterms:title, dcterms:description, dcterms:license, prov:wasGeneratedBy.
  • RASTER: above + geo:hasGeometry (footprint), dct:spatial, dcterms:temporal, si:bands (list of band descriptors), si:resolution.
  • ARRAY: above + si:variables (list with shape/dtype/dim labels).
  • TILES: above + si:zoomRange, si:tileScheme. Writers validate; missing required triples → warning (v1) or error (v2).
  • B. Soft shape — recommend but don't enforce. Document the per-kind profiles; discovery layer copes with absence.
  • C. Defer entirely to a follow-up spec. Ship the envelope, design the discovery shape later.

Recommendation: A, in a separate follow-up spec written before the first non-tabular handler lands. The envelope can ship without it, but no GeoTIFF handler should ship without the RDF shape it must emit. Treat as a gating dependency for non-tabular handlers, not for the envelope itself.

16. RDF value typing (IRI vs literal vs blank node)

Problem. custom_properties: dict[str, Any] can't tell whether the value is an IRI reference, a typed literal (xsd:date, xsd:double), a language-tagged string, or a blank node. Discovery queries need this distinction.

Options.

  • A. JSON-LD value conventions. Values are either plain Python literals (str, int, float, bool, datetime) or dicts following JSON-LD: {"@id": "..."} for IRIs, {"@value": "...", "@type": "xsd:date"} for typed literals, {"@value": "...", "@language": "en"} for language tags.
  • B. Small RDF term wrappers. IRI("..."), Literal("...", lang="en"). Serialised to JSON-LD on emit.
  • C. Free-form; rely on downstream interpretation.

Recommendation: A. JSON-LD conventions are the lingua franca; users (and tooling) already understand them. No new types to import.

17. Stable asset @id for cross-project discovery

Problem. slug is project-local. Two unrelated projects both have current-un-members. Discovery across packages needs globally stable IDs.

Options.

  • A. Asset @id derived from package + slug + version. E.g., https://sunstone.institute/datasets/sunstoneinstitute/sunstone-py/current-un-members@1.0.0. Publishable when the package publishes.
  • B. Content-hash identity. @id is urn:sunstone:sha256:.... Stable across rebuilds when content is identical.
  • C. Both — @id is package-namespaced; si:dataHash is content-hash; discovery uses either.

Recommendation: C. Two-axis identity: humans use namespaced names, machines use content hashes, both work.

18. Per-band / per-variable / per-layer component metadata

Problem. LineageMetadata.field_derivations is tabular-specific. Raster bands, NPZ variables, and tile layers all want per-component metadata (units, dtype, description, derivation). Repurposing field_derivations to mean "band" sometimes is confusing.

Options.

  • A. Introduce component_metadata: dict[str, ComponentSchema] on Metadata, keyed by component name. ComponentSchema is a neutral structure with name, kind (column/band/variable/layer), dtype, units, description, derived_from. field_metadata becomes a tabular-flavoured view onto it.
  • B. Per-kind metadata classes. BandMetadata, VariableMetadata, etc., on Asset.extras. Metadata stays tabular-only for fields.
  • C. Defer. Ship without per-component metadata for non-tabular kinds; revisit when a handler needs it.

Recommendation: A. A neutral component model future-proofs the discovery layer and avoids per-kind sharding. Migration: field_metadata keeps working as a typed view; new kinds populate component_metadata directly.

19. Deprecation warning timing

Problem. Emitting DeprecationWarning at handler-registration time breaks users running with warnings-as-errors during import.

Options.

  • A. Emit on first actual use, once per handler class. Register a sentinel; first read()/write() call through the adapter emits the warning, sets the sentinel.
  • B. Emit on registration but use PendingDeprecationWarning until a configurable release. Promote to DeprecationWarning one minor before removal.
  • C. Environment-variable escape hatch. SUNSTONE_SUPPRESS_LEGACY_WARNINGS=1 disables.

Recommendation: A + C. First-use timing avoids import-time failures; the env-var escape hatch is cheap insurance. B is fine too but more bookkeeping.

How to respond

For each decision, leave an inline crit comment with one of:

  • "go" — accept the recommendation as written.
  • "do X instead" — pick a different option (A/B/C/...) with one-line reason if non-obvious.
  • "defer" — push to a later spec; tell me where it should land (parent spec follow-up, separate spec, never).
  • "reframe" — the question itself is wrong; here's the right question.

I'll fold the decisions back into the parent spec in one pass, then we ship.