Skip to content

Core Concepts

Understanding the key concepts behind sunstone-py's data management and lineage tracking.

Asset Envelope

Everything sunstone reads or writes is an Asset — a uniform container with four fields:

  • payload — the kind-native data: a pandas.DataFrame, a NumPy ndarray, a dict[str, ndarray], a tile pyramid descriptor, or raw bytes for opaque binaries.
  • kind — an AssetKind enum value (TABULAR, RASTER, ARRAY, TILES, BLOB).
  • metadata — the unified Metadata container (lineage, per-component schema, RDF properties, license).
  • extras — kind-specific accessory info (rasterio profile, CRS, chunk spec).

Two equivalent entry points produce and consume assets:

# Kind-agnostic — works for any AssetKind
asset = sunstone.read('inputs/era5_2024.zarr')
sunstone.write(asset.derive(new_payload, slug='child', name='Child'),
               'outputs/era5_derived.zarr')

# Tabular shortcut — pandas-like API
from sunstone import pandas as pd
df = pd.read_csv('inputs/schools.csv')
df.to_csv('outputs/summary.csv', slug='summary', name='Summary')

sunstone.DataFrame is a thin facade over an Asset of kind=AssetKind.TABULAR; the two paths record identical lineage. Use the pandas-style API for code that should read like pandas, and the Asset API for code that needs to be uniform across kinds.

Each kind has its own quickstart:

  • Tabular — pandas.DataFrame payloads (CSV, Parquet, Excel, JSON, TSV).
  • Tensors — dict[str, ndarray] payloads (.npz, Zarr, HDF5/NetCDF-4).
  • Images — single-ndarray rasters (GeoTIFF on the roadmap).
  • Tile pyramids — pre-tiled multi-resolution data.
  • Blob / opaque binaries — bytes payloads for PDF, DOCX, PPTX, RTF, plain-text reports, and similar artefacts sunstone ingests without interpreting.

Asset.derive(payload, ..., derived_from=[...]) produces a child asset and records prov:wasDerivedFrom for each parent. Slug and name do not inherit — the child is a new dataset.

Pandas-Like API

sunstone-py provides a drop-in replacement for pandas that adds lineage tracking:

import sunstone
from sunstone import pandas as pd

# Configure the default project path once
sunstone.set_project_path('/path/to/project')

# Works like pandas, but tracks lineage
df = pd.read_csv('input.csv')
df2 = pd.read_csv('input2.csv')

# All pandas operations work
filtered = df[df['value'] > 100]
grouped = df.groupby('category').sum()

# Merge/join operations combine lineage from both sources
merged = pd.merge(df, df2, on='key')
concatenated = pd.concat([df, df2])

Key Differences from Plain Pandas

  1. Project path configuration: Reads resolve paths against a project directory containing datasets.yaml. Set it once with sunstone.set_project_path(...), pass project_path= per call, or rely on the Path.cwd() fallback. Use with sunstone.use_project_path(...): for a scoped override.
  2. Dataset registration: All reads and writes must correspond to entries in datasets.yaml
  3. Access underlying data: Use .data to access the pandas DataFrame directly
  4. Save with metadata: Write operations require slug and name for dataset registration

Strict vs Relaxed Mode

sunstone-py operates in two modes that control how it interacts with datasets.yaml:

Relaxed Mode (Default)

Behavior: - Writing to new outputs auto-registers them in datasets.yaml - More flexible for exploratory work - Schema is inferred from the data - Lineage metadata is automatically added

Use when: - Doing exploratory data analysis - Prototyping new analyses - Working in notebooks - Iterating quickly on data transformations

Example: 

# This will auto-create an entry in datasets.yaml
result.to_csv(
    'outputs/new-analysis.csv',
    slug='new-analysis',
    name='New Analysis Results',
    index=False
)

Strict Mode

Behavior: - All reads and writes must be pre-registered in datasets.yaml - Raises StrictModeError if dataset not found - Ensures complete documentation of data operations - Validates that lineage matches what's recorded

Use when: - Running production pipelines - Need reproducibility guarantees - Working in team environments - Preparing for publication or sharing

Enable strict mode:

# Per-operation
df = pd.read_csv('data.csv', strict=True)

# Globally via environment variable
import os
os.environ['SUNSTONE_DATAFRAME_STRICT'] = '1'

# Via CLI for entire dataset
# sunstone dataset lock my-dataset

Example error in strict mode: 

# This will raise StrictModeError if 'new-output' not in datasets.yaml
result.to_csv(
    'outputs/new-output.csv',
    slug='new-output',
    name='New Output',
    index=False,
    strict=True
)

Lineage Tracking (W3C PROV-O)

Lineage tracking automatically captures the provenance of your data through all operations. Since v1.5.0, the lineage model is aligned with W3C PROV-O, the standard ontology for provenance.

PROV-O Concepts

sunstone-py maps its data model to PROV-O:

  • Entity: A dataset (DatasetMetadata) — the thing being tracked
  • Activity: A script or notebook execution (Activity) — the process that transforms data
  • Agent: A user, organization, or software (Agent) — who is responsible

What Gets Tracked

Sources: - Input datasets that were read - Their metadata (slug, name, location) - License information - Source attribution (as PROV-O Agents)

Activities: - Script/notebook executions with timestamps - Which entities were used (with column-level detail) - Which entities were generated - Associated agents (user, software) - Git commit hash and dirty state

Field Derivations: - Which output columns came from which source datasets - Column-level provenance (prov:qualifiedDerivation) - Auto-populated on read so provenance flows through merge/join/concat

Metadata: - Content hash (detects when data actually changes) - Creation timestamp (only updated when content changes) - Source relationships

Accessing Lineage

# Read and transform data
df = pd.read_csv('input.csv', project_path=PROJECT_PATH)
result = df[df['value'] > 100].groupby('category').sum()

# Access lineage through metadata (preferred)
print(result.metadata.lineage.sources)
print(result.metadata.lineage.get_licenses())

# Check field derivations
if result.metadata.lineage.field_derivations:
    for fd in result.metadata.lineage.field_derivations:
        print(f"  {fd.output_field} <- {fd.source_entity}.{fd.source_field}")

# Check activity details
if result.metadata.lineage.activity:
    activity = result.metadata.lineage.activity
    print(f"Activity: {activity.id}")
    print(f"Started: {activity.started_at}")
    for agent in activity.was_associated_with:
        print(f"Agent: {agent.label} ({agent.type.value})")

Field-Level Derivations

When you read a dataset, sunstone automatically records which columns came from which source. This propagates through operations:

# Read: each column gets a derivation record
schools = pd.read_csv('schools.csv', project_path=PROJECT_PATH)
# schools.metadata.lineage.field_derivations contains:
#   FieldDerivation(output_field='name', source_entity='school-data', source_field='name')
#   FieldDerivation(output_field='enrollment', source_entity='school-data', source_field='enrollment')
#   ...

# Merge: derivations from both sources are combined
merged = pd.merge(schools, teachers, on='school_id')
# merged has derivations from both 'school-data' and 'teacher-data'

Lineage Persistence

When you save a DataFrame, lineage is automatically written to datasets.yaml, including PROV-O activity tracking:

result.to_csv(
    'outputs/summary.csv',
    slug='summary',
    name='Summary Results',
    index=False
)

This adds to datasets.yaml:

outputs:
  - name: Summary Results
    slug: summary
    location: outputs/summary.csv
    fields:
      - name: category
        type: string
      - name: value
        type: number
    lineage:
      content_hash: abc123...
      created_at: '2026-02-04T10:30:00'
      sources:
        - slug: input-data
      activity:
        id: exec-20260204T103000-abc123
        agents:
          - id: stig
            type: prov:Person
          - id: sunstone-py
            type: prov:SoftwareAgent
            version: '1.5.0'
        used:
          - entity: input-data
        started_at: '2026-02-04T10:29:55'
        ended_at: '2026-02-04T10:30:00'
      field_derivations:
        - output_field: category
          source_entity: input-data
          source_field: category
        - output_field: value
          source_entity: input-data
          source_field: value

Lineage Propagation

Lineage automatically propagates through operations:

# Read two sources
schools = pd.read_csv('schools.csv', project_path=PROJECT_PATH)  # source 1
teachers = pd.read_csv('teachers.csv', project_path=PROJECT_PATH)  # source 2

# Merge combines lineage from both
merged = pd.merge(schools, teachers, on='school_id')

# Result tracks both sources
print(len(merged.metadata.lineage.sources))  # 2

Content Hash Optimization

The content hash prevents unnecessary timestamp updates:

# First save
result.to_csv('output.csv', slug='output', name='Output')
# lineage.created_at = '2026-02-04T10:00:00'

# Re-run with same result
result.to_csv('output.csv', slug='output', name='Output')
# lineage.created_at = '2026-02-04T10:00:00'  (unchanged!)

# Re-run with different result
result_v2.to_csv('output.csv', slug='output', name='Output')
# lineage.created_at = '2026-02-04T11:00:00'  (updated!)

DataFrame Metadata

Every DataFrame carries a metadata container that holds lineage and per-field annotations. This metadata flows through operations and is persisted to datasets.yaml on write.

The Metadata Container

df = pd.read_csv('data.csv', project_path=PROJECT_PATH)

df.metadata.slug = 'my-analysis'
df.metadata.name = 'My Analysis'
df.metadata.description = 'Analysis of school enrollment data'

# RDF prefixes and custom properties
df.metadata.rdf_prefixes = {'schema': 'http://schema.org/'}
df.metadata.custom_properties = {'schema:about': 'Education'}

# Lineage is accessed through metadata
print(df.metadata.lineage.sources)

Per-Field Metadata

Annotate individual columns with descriptions, units, and source tracking:

df.set_field_metadata('enrollment', description='Total enrolled students', unit='students')
df.set_field_metadata('area_km2', description='School district area', unit='km^2')
df.set_field_metadata('density', description='Students per square kilometer', unit='students / km^2')

Field metadata is written to datasets.yaml alongside the field schema:

fields:
  - name: enrollment
    type: integer
    description: Total enrolled students
    unit: students
  - name: area_km2
    type: number
    description: School district area
    unit: km^2

Deprecation: df.lineage

The old df.lineage accessor still works but is deprecated. Use df.metadata.lineage instead.

Plugin System

Reading, writing, and URL fetching are handled by a plugin registry. Built-in handlers cover common formats and URL schemes; external plugins are discovered automatically via entry points.

Built-in Support

Formats: CSV, JSON, Excel, Parquet, TSV

URL schemes: - Local file paths (built-in) - http:// and https:// (built-in, with SSRF protection) - gs:// (requires sunstone-py[gcs]) - s3:// and r2:// (requires sunstone-py[s3])

Using the Plugin Registry

from sunstone.plugins import PluginRegistry
from pathlib import Path

registry = PluginRegistry.get('/path/to/project')

# Fetch a file from any supported URL
registry.fetch('gs://my-bucket/data.csv', Path('data/local.csv'))

Writing Custom Plugins

Implement one or more plugin protocols (AuthProvider, URLHandler, FormatHandler) and register via entry points:

# In your plugin's pyproject.toml
[project.entry-points."sunstone.plugins"]
my-plugin = "my_package:MyPlugin"

Plugin Configuration

Configuration uses cascading precedence:

  1. datasets.yaml → plugins.<name> section (highest priority)
  2. pyproject.toml → [tool.sunstone.plugins.<name>]
  3. Environment variables → SUNSTONE_PLUGIN_<NAME>_<KEY>

Unit-Aware Arithmetic

sunstone-py integrates with Pint for unit-aware column operations. When columns have units set via set_field_metadata(), arithmetic operations validate unit compatibility.

Unit Modes

export SUNSTONE_UNIT_MODE=auto  # or: strict, relaxed (default)

from sunstone.units import set_unit_mode
set_unit_mode('auto')
Mode Behavior
relaxed No unit validation (default)
strict Raises UnitError on incompatible operations
auto Auto-converts compatible units, warns on mismatch

Example

from sunstone.units import set_unit_mode
set_unit_mode('auto')

df.set_field_metadata('distance_km', unit='km')
df.set_field_metadata('distance_miles', unit='mile')

# In auto mode, merging DataFrames with km and miles on the same
# column will automatically convert to a common unit

QUDT Round-Tripping

Units stored as QUDT URIs in datasets.yaml are preserved through read/write cycles. The original URI is stored in FieldSchema.unit_source so it round-trips without loss.

License Compatibility

sunstone-py treats licenses as a first-class part of lineage: every input declares a source.license, every output declares an effective license (its own, its packages[] entry, or the top-level package.license), and the compatibility between them is checked automatically at write time.

There are three layers of license handling, intentionally graded from soft to hard:

  1. Lint (sunstone lint) — R005 flags a dataset that has no license at all; R006 flags a license string that isn't a recognised SPDX identifier or LicenseRef-* form.
  2. CLI audit (sunstone license check) — runs the compatibility engine across every output's wasDerivedFrom chain and exits non-zero on conflict. Use it in CI. See the CLI Guide.
  3. Write-time enforcement — to_csv() / to_parquet() raise LicenseCompatibilityError when the target license is incompatible with a source license in the current session lineage. When no target license is declared, one is auto-derived from the sources (inheriting a single source's license, or picking the most restrictive license that satisfies all sources) and persisted to datasets.yaml. Pass check_license=False to skip on a specific write.

How the Compatibility Engine Works

Each license in the embedded registry is described by four boolean properties — public_domain, attribution, share_alike, non_commercial — plus a family tag for ShareAlike. A (source → target) pair is checked against this rule set:

Source property Constraint on target
public_domain None — compatible with anything.
share_alike Target must also be share_alike and share the same family. (CC-BY-SA-4.0 and CC-BY-SA-3.0 are different families on purpose — Creative Commons treats them as one-way upgradable, not mutually equivalent.)
non_commercial Target must also be non_commercial.
attribution (without SA/NC) Target must be attribution or share_alike (SA implies attribution).

A check fails as soon as any source/target pair violates the rules; the resulting LicenseCompatibilityResult lists every conflict and a suggestions list of registry licenses that would satisfy all known sources. The most-restrictive ordering — used by get_most_restrictive_license() and by suggestion ranking — is ShareAlike > NonCommercial > Attribution > Public Domain.

Embedded License Registry

The registry is intentionally focused on common research-data licenses. Anything outside it must use a LicenseRef-* identifier; unregistered LicenseRef-* ids pass lint R006 but are reported as unknown_sources by the compatibility engine and excluded from the rule check.

SPDX id Class
CC0-1.0, PDDL-1.0, LicenseRef-US-PD Public domain
CC-BY-4.0, CC-BY-3.0, CC-BY-3.0-IGO Attribution only
ODC-By-1.0, LicenseRef-OGL-3.0, NLOD-1.0, NLOD-2.0 Attribution only
CC-BY-SA-4.0, CC-BY-SA-3.0, ODbL-1.0 Attribution + ShareAlike (each is its own family)
CC-BY-NC-4.0, CC-BY-NC-3.0-IGO Attribution + NonCommercial
CC-BY-NC-SA-4.0, CC-BY-NC-SA-3.0, CC-BY-NC-SA-3.0-IGO Attribution + NonCommercial + ShareAlike (each is its own family)

Adding a license is a one-line append to _REGISTRY_ENTRIES in src/sunstone/licenses.py. The full set is also available programmatically via sunstone.licenses.known_licenses().

Worked Example

A source un-members is licensed CC-BY-NC-4.0. You merge it with a CC-BY-4.0 input and try to publish under CC-BY-4.0:

result.to_csv(
    "outputs/derived.csv",
    slug="derived",
    name="Derived Data",
    license="CC-BY-4.0",
)
# raises LicenseCompatibilityError:
#   CC-BY-NC-4.0 is NonCommercial: derivatives must also be NonCommercial, not CC-BY-4.0
#   Suggested compatible target licenses: CC-BY-NC-4.0, CC-BY-NC-SA-4.0, CC-BY-NC-3.0-IGO

The fix is to widen the output to a NonCommercial license — or, if you have rights to do so, to remove the NC source from the lineage. The engine does not let you "wash out" a NonCommercial source by mixing it with a permissive one; that is the correct behaviour under Creative Commons.

Auto-Derived Target Licenses

When the output has source licenses but no declared target (no license: on the dataset, no packages[].license / package.license covering it, and no license= argument to the writer), the writer derives a target automatically:

  • One unique source license — the output inherits it.
  • Multiple compatible source licenses — the writer picks the most restrictive license that satisfies every source (e.g., CC-BY-4.0 + CC-BY-NC-4.0 → CC-BY-NC-4.0).
  • Mutually incompatible sources, or unknown licenses among multiples — LicenseCompatibilityError. The writer refuses to guess; declare an explicit license: to proceed.

Any auto-derived license is persisted to datasets.yaml so future reads and sunstone license check runs see it as declared.

Conflicting Licenses

An explicitly declared target license that is incompatible with a source license is always a hard failure (LicenseCompatibilityError) — silently publishing under the wrong license can be a legal problem. Combine sunstone lint --warnings-as-errors (catches missing licenses at lint time) with sunstone license check (catches conflicts) in CI to make both classes hard failures before publish.

Dataset Metadata

Every dataset in datasets.yaml has rich metadata:

Required Fields

- name: Human-Readable Name
  slug: machine-readable-slug
  location: path/to/file.csv
  fields:
    - name: column_name
      type: string  # or number, integer, boolean, date, datetime

Optional Fields

# Top-level publishing configuration (applies to all outputs)
publish:
  enabled: true
  to: gs://bucket-name/path/
  flatten: false  # optional: flatten directory structure

inputs:
  - name: Example Dataset
    slug: example
    location: data/example.csv

    # Source attribution for inputs
    source:
      name: Data Provider Name
      location:
        data: https://example.com/data.csv
      attributedTo: Organization or Person
      acquiredAt: '2025-01-15'
      acquisitionMethod: manual-download  # or api, web-scraping, etc.
      license: CC-BY-4.0

    # Strict mode flag
    strict: true

outputs:
  - name: Output Dataset
    slug: output-example
    location: outputs/example.csv

    # Strict mode flag
    strict: true

    # Lineage metadata (auto-generated)
    lineage:
      content_hash: abc123...
      created_at: '2026-02-04T10:00:00'
      sources:
        - slug: source-dataset

Publishing Configuration

The top-level publish section controls how data packages are published:

publish:
  enabled: true                              # Required: enable publishing
  to: gs://bucket/datasets/project-name/     # GCS upload destination
  as: https://data.example.com/project-name/ # Optional: public URL base for datapackage.json
  flatten: false                             # Optional: flatten directory structure

Path Resolution:

  • If to ends with .json: Used as the datapackage filename
  • gs://bucket/countries.json → datapackage at gs://bucket/countries.json

  • Data files in gs://bucket/ (relative to datapackage directory)

  • If to doesn't end with .json: Treated as a directory

  • gs://bucket/datasets/project/ → datapackage at gs://bucket/datasets/project/datapackage.json
  • Data files in gs://bucket/datasets/project/

Public URL Option (as):

When your GCS bucket is served via a CDN or custom domain, use as to set the public-facing URLs in datapackage.json:

publish:
  to: gs://my-bucket/datasets/project/      # Files uploaded here
  as: https://data.example.com/project/     # URLs in datapackage.json use this base
  • Files are uploaded to gs://my-bucket/datasets/project/outputs/data.csv
  • But datapackage.json contains: "path": "https://data.example.com/project/outputs/data.csv"

This allows data consumers to fetch files directly from your public URL.

Flatten Option:

  • flatten: false (default): Preserves directory structure from location field

  • location: outputs/data/file.csv → gs://bucket/project/outputs/data/file.csv

  • flatten: true: Puts all files in same directory as datapackage.json

  • location: outputs/data/file.csv → gs://bucket/project/file.csv

Validation Tools

Check notebooks and scripts for correct import usage:

import sunstone

# Check a single notebook
result = sunstone.check_notebook_imports('analysis.ipynb')
print(result.summary())

# Check all notebooks in project
results = sunstone.validate_project_notebooks('/path/to/project')
for path, result in results.items():
    if not result.is_valid:
        print(f"\n{path}:")
        print(result.summary())

What validation checks: - Files use from sunstone import pandas as pd instead of plain pandas - No direct pandas imports in data processing code - Proper usage of project_path parameter

Project Path Configuration

read_csv, read_excel, read_json, read_dataset, and the DataFrame constructor all need to know which project's datasets.yaml to consult. They accept a project_path= argument; when none is given, they fall back to a process-wide default, and finally to Path.cwd().

import sunstone
from sunstone import pandas as pd
from pathlib import Path

# 1. Set once for the entire process (recommended in notebooks/scripts)
sunstone.set_project_path(Path(__file__).parent)
df = pd.read_csv('inputs/data.csv')          # uses configured path

# 2. Pass explicitly per call (overrides the default)
df = pd.read_csv('inputs/data.csv', project_path='/other/project')

# 3. Scoped override
with sunstone.use_project_path('/temporary/project'):
    df = pd.read_csv('inputs/other.csv')
# default is restored after the with block

# Read or clear the configured default
print(sunstone.get_project_path())
sunstone.clear_project_path()

The configured value is held in a contextvars.ContextVar, so it is safe across threads and async tasks.

Organizing Datasets Across Files

For projects with many datasets, datasets.yaml can pull in additional files via an include: directive:

# datasets.yaml
package:
  name: my-project
  license: CC-BY-4.0

include:
  - datasets/inputs.yaml
  - datasets/outputs.yaml

# Top-level inputs/outputs/packages here are still allowed
inputs: []
outputs: []

Each included file may declare its own inputs:, outputs:, and packages: lists. Top-level configuration that affects the whole project (defaults, rdfPrefixes, package, publish, min_sunstone_version) is not allowed in included files — only in the main datasets.yaml.

Other rules:

  • Includes are not nested — an included file cannot itself contain an include: key.
  • Slug collisions across files are an error: every dataset slug and every package name must be unique across the merged set.
  • Paths are relative to the file containing the include: directive.

Environment Variables

SUNSTONE_DATAFRAME_STRICT

Enable strict mode globally:

export SUNSTONE_DATAFRAME_STRICT=1
# or
export SUNSTONE_DATAFRAME_STRICT=true

# Now all operations are strict by default
df = pd.read_csv('input.csv')  # strict=True implied

Best Practices

Start Relaxed, Lock for Production

  1. Development: Use relaxed mode for exploration
  2. Refinement: Review auto-generated datasets.yaml entries
  3. Production: Lock datasets with sunstone dataset lock

Lint Before You Ship

Run sunstone lint to catch missing licenses, units, descriptions, and slugs that don't follow project conventions before publishing. In CI, use sunstone lint --warnings-as-errors to make recommended-but-not-required metadata mandatory. See the CLI Guide for the full rule list.

Document Sources Thoroughly

inputs:
  - name: UN Member States
    slug: un-members
    location: inputs/un_members.csv
    source:
      name: United Nations
      location:
        data: https://www.un.org/en/about-us/member-states
      attributedTo: United Nations
      acquiredAt: '2025-01-15'
      acquisitionMethod: manual-download
      license: Public Domain
      notes: |
        Downloaded from the official UN website.
        Data accurate as of January 2025.

Use Descriptive Slugs

# Good
slug: school-enrollment-by-district
slug: teacher-demographics-2025

# Avoid
slug: data1
slug: output
slug: final_final_v2

Track Operations Explicitly

# Instead of chaining without description
result = df.apply(complex_function)

# Add operation description for lineage
result = df.apply_operation(
    complex_function,
    description="Apply enrollment adjustment factors"
)

Version Your Datasets

outputs:
  - name: School Summary v2.1
    slug: school-summary-v2-1
    location: outputs/school-summary-v2.1.csv

Or use the version field in package metadata when building data packages.