CLI Guide¶

The sunstone command-line interface provides tools for managing datasets and building data packages.

Installation¶

The CLI is automatically installed with sunstone-py:

uv add sunstone-py

# Verify installation
sunstone --version

Environment Commands¶

sunstone env manages cascading TOML configuration that overlays values on os.environ for CLI invocations and Python sessions that call sunstone.activate_environment(). Environments are useful for storing catalog URLs, warehouse names, region settings, and op:// references to secrets without committing them to datasets.yaml.

Config file precedence (highest wins for active-environment selection):

SUNSTONE_DATA_ENV env var.
.sunstone/data_platform.toml (project — walked up from cwd).
~/.config/sunstone/data_platform.toml (user).
/etc/sunstone/data_platform.toml (system).

Within an environment definition, field-level merging follows the same order — project entries override user entries override system entries.

Show Active Environment¶

sunstone env

Lists all defined environments and marks the active one. Output:

Active: dev (from /Users/me/.config/sunstone/data_platform.toml)

* dev          3 keys, sections: data-platform   (/Users/me/.config/sunstone/data_platform.toml)
  prod         2 keys                            (/etc/sunstone/data_platform.toml)

Switch Active Environment¶

# Set the active environment in the project file
sunstone env use dev

# Set it in the user file instead
sunstone env use dev --user

Add an Environment¶

KEY=VAL entries with a . in the key are written to plugin-namespaced subtables — handy for per-plugin configuration like data-platform.warehouse=main.

# User scope (default)
sunstone env add dev CATALOG_URL=https://data.dev.example.com

# Mix top-level scalars and a subtable entry
sunstone env add dev data-platform.warehouse=main GIT_BRANCH=main

# Write to project or system scope instead
sunstone env add dev CATALOG_URL=https://... --scope project
sunstone env add dev CATALOG_URL=https://... --scope system

# Replace an existing entry
sunstone env add dev CATALOG_URL=https://... --overwrite

--scope accepts user (default), project, or system and applies to add, set, unset, and remove.

Update an Environment¶

env set merges entries into an existing environment without touching unspecified keys.

sunstone env set dev CATALOG_URL=https://new.example.com
sunstone env set dev data-platform.warehouse=staging --scope project

If the same environment is also defined in a higher-precedence layer, the CLI warns that the update will be shadowed.

Remove Entries¶

# Remove specific keys (dotted keys target subtables; the subtable is
# deleted if it ends up empty)
sunstone env unset dev CATALOG_URL data-platform.warehouse

# Remove the entire environment
sunstone env remove dev
sunstone env remove dev --scope system

Credential References¶

Values of the form op://vault/item/field are resolved through 1Password CLI at activation time, so secrets stay out of version control. Real env vars still win — if CATALOG_URL is already set in the shell, the environment value is left alone.

Dataset Commands¶

List Datasets¶

Show all input and output datasets in your project:

sunstone dataset list
sunstone dataset list -f path/to/datasets.yaml

Example output:

Inputs:
  - school-data (School Data)
  - teacher-data (Teacher Information)

Outputs:
  - school-summary (School Enrollment Summary) [publish]
  - analysis-results (Analysis Results) [strict, publish]

Validate Datasets¶

Check that your datasets.yaml follows the correct structure:

# Validate all datasets
sunstone dataset validate

# Validate specific datasets
sunstone dataset validate school-data summary-data

# Validate with custom file location
sunstone dataset validate -f path/to/datasets.yaml

Validation checks:

Required fields (name, slug, location, fields)
Valid field types (string, number, integer, boolean, date, datetime)
Duplicate slugs
Proper YAML structure

Example output:

✓ datasets.yaml is valid

Example error:

Validation errors:
  - outputs[0]: missing required field 'fields'
  - inputs[1].fields[2]: invalid type 'text' (must be one of: string, number, integer, boolean, date, datetime)
  - Dataset 'school-data' not found

Lock Datasets (Enable Strict Mode)¶

Enable strict mode for datasets to prevent programmatic modifications:

# Lock specific datasets
sunstone dataset lock school-data summary-data

# Lock all datasets
sunstone dataset lock

Output:

✓ Locked 2 dataset(s): school-data, summary-data

When a dataset is locked, any attempt to modify it in datasets.yaml will fail with an error. This ensures complete documentation of all data operations.

Unlock Datasets (Disable Strict Mode)¶

Disable strict mode to allow programmatic modifications:

# Unlock specific datasets
sunstone dataset unlock school-data

# Unlock all datasets
sunstone dataset unlock

Output:

✓ Unlocked 1 dataset(s): school-data

Lint Command¶

sunstone lint checks datasets.yaml against the Sunstone Minimum Viable Metadata recommendations. It complements sunstone dataset validate (which checks structure and types) by flagging missing metadata that hurts reproducibility and discoverability.

# Lint the current project
sunstone lint

# Lint a project at a specific path
sunstone lint -p path/to/project

# Use a non-default datasets file
sunstone lint -f config/my-datasets.yaml

# Run only specific rules
sunstone lint --rules R005,R104

# Treat warnings as errors (useful in CI)
sunstone lint --warnings-as-errors

# Machine-readable output
sunstone lint --json

Rules¶

ID	Severity	Title
R001	error	Dataset missing `name`
R002	error	Dataset missing `slug`
R003	error	Dataset missing `location`
R004	error	Dataset missing `description`
R005	error	Dataset missing license (input: `source.license`; output: dataset, `package.license`, or matching `packages[]` entry)
R006	error	License is not a recognised SPDX or allow-list identifier
R007	error	Field missing `name`
R008	error	Field missing `type`
R009	error	Malformed `lint.disable` entry (cannot itself be suppressed)
R101	warning	Input missing `source` block
R102	warning	Source block malformed or missing required keys
R103	warning	Numeric field missing `unit`
R104	warning	Slug not in kebab-case
R105	warning	Published output field missing `description`
R201	info	Generic field name (`total`, `value`, ...) without a substantive description
R202	info	Generic dataset name (`data`, `output`, ...)

R001–R009 are errors (exit non-zero), R101–R105 are warnings (exit zero unless --warnings-as-errors), R201–R202 are informational.

Suppressing Rules¶

Specific findings can be suppressed in datasets.yaml with a written justification:

lint:
  disable:
    R104: "Slug mirrors the upstream UN identifier 'A_HRC_RES'"
    R103: "Pure-count column, unit would be misleading"

Suppressed findings stay in the report under a separate suppressed list so reviewers can audit the reasons later. R009 itself cannot be suppressed — it's the rule that catches malformed suppressions.

Programmatic Use¶

from sunstone import lint_project

report = lint_project('/path/to/project')

if report.errors:
    for v in report.errors:
        print(f"{v.rule_id} {v.location}: {v.message}")

# Suppressed findings (still tracked for audit)
for v in report.suppressed:
    reason = report.suppressions.get(v.rule_id, "")
    print(f"  [{v.rule_id}] suppressed because: {reason}")

Example Output¶

[R005] ERROR inputs[0].source.license: missing license
    hint: Add an SPDX license identifier (e.g. 'CC-BY-4.0', 'MIT').
[R103] WARNING outputs[0].fields[2]: numeric field 'enrollment' has no unit
    hint: Add a 'unit:' (e.g. 'meter', 'USD'), or accept the lock-file unit if derived from arithmetic.
[R104] WARNING inputs[1].slug: slug 'School_Data' is not kebab-case
    hint: Use lowercase ASCII letters/digits separated by single hyphens.

Suppressed by lint.disable (1):
  [R201] outputs[0].fields[0]: generic field name 'value' without a substantive description
      reason: Column name fixed by upstream contract

Summary: 1 error(s), 2 warning(s), 0 info, 1 suppressed

Package Commands¶

Build Data Package¶

Create a datapackage.json from your datasets.yaml:

# Build with default output (datapackage.json)
sunstone package build

# Specify custom output file
sunstone package build -o path/to/package.json

# Use custom datasets file
sunstone package build -f path/to/datasets.yaml -o package.json

This creates a Data Package v2 with all publishable output datasets as resources.

Example output:

  + school-summary
  + analysis-results

✓ Created datapackage.json with 2 resource(s)

Requirements: - Only output datasets with publish.enabled: true are included - Output files must exist at their specified locations - Requires frictionless package to be installed

Push to Google Cloud Storage¶

Upload your data package and all output datasets to GCS.

Prerequisites:

Pushing to gs:// requires the optional GCS extra (s3:///r2:// need [s3]):

uv add 'sunstone-py[gcs]'

Without it, the push fails with "No URL handler found for: gs://…".

Publishing requires a top-level publish configuration in datasets.yaml:

publish:
  enabled: true
  to: gs://my-bucket/datasets/project-name/
  as: https://data.example.com/project-name/  # optional: public URL base
  flatten: false  # optional, default: false

Commands:

# Push to configured destination
sunstone package push

# Push to environment-specific destination (if publish.to not set)
sunstone package push --env prod

# Override destination
sunstone package push -d gs://my-bucket/datasets/project-name/

# Use custom datasets file
sunstone package push -f path/to/datasets.yaml

Path Resolution:

The publish.to field determines where files are uploaded:

Directory path (no .json extension):

publish:
  to: gs://bucket/datasets/countries/

Uploads to:

gs://bucket/datasets/countries/datapackage.json
gs://bucket/datasets/countries/outputs/data.csv

Custom datapackage filename (ends with .json):

publish:
  to: gs://bucket/datasets/countries.json

Uploads to:

gs://bucket/datasets/countries.json
gs://bucket/datasets/outputs/data.csv (relative to datapackage directory)

Flattened structure (ignores subdirectories in location):

publish:
  to: gs://bucket/datasets/countries/
  flatten: true

Uploads to:

gs://bucket/datasets/countries/datapackage.json
gs://bucket/datasets/countries/data.csv (no outputs/ prefix)

Public URL mapping (different URLs in datapackage.json vs upload destination):

publish:
  to: gs://bucket/datasets/countries/
  as: https://data.example.com/countries/

Uploads to GCS:

gs://bucket/datasets/countries/datapackage.json
gs://bucket/datasets/countries/outputs/data.csv

But datapackage.json contains public URLs:

{
  "resources": [{
    "path": "https://data.example.com/countries/outputs/data.csv"
  }]
}

This is useful when your GCS bucket is served via a CDN or custom domain.

Environment variable expansion:

Destination URLs support ${VAR} or ${VAR:-default} syntax:

publish:
  to: gs://${BUCKET:-default-bucket}/datasets/${PROJECT}/

Or via command line:

sunstone package push -d "gs://${BUCKET}/datasets/${PROJECT}/"

Example output:

✓ Uploaded datasets/countries/datapackage.json
✓ Uploaded outputs/current_countries.csv

✓ Package pushed to: gs://my-bucket/datasets/countries/

License Commands¶

sunstone license inspects and audits the licenses declared in datasets.yaml. Use it alongside the lint rules R005 (missing license) and R006 (unrecognised SPDX identifier) — lint catches missing and malformed licenses, the license subcommand catches incompatible ones.

The compatibility engine is rules-based and consults an embedded registry of common research-data licenses (CC family, CC0, ODC-By, ODbL, PDDL, OGL-3.0, NLOD, and US-PD LicenseRef-* entries). See Concepts → License Compatibility for the rule reference.

List Licenses¶

Show every license referenced in the project and which datasets declare it:

# Default datasets.yaml in current directory
sunstone license list

# Custom file or project directory
sunstone license list -f path/to/datasets.yaml

# Machine-readable output
sunstone license list --json

Example output:

CC-BY-4.0
  - input:un-members
  - input:world-bank-gdp
CC-BY-SA-4.0
  - output:enriched-members

Output licenses are resolved to their effective license — explicit license: on the dataset, otherwise the matching packages[] entry, otherwise the top-level package.license.

Check License Compatibility¶

Verify that each output's declared license is compatible with the licenses of every source dataset in its wasDerivedFrom chain:

# Check every output
sunstone license check

# Check a single output
sunstone license check enriched-members

# Machine-readable output (for CI)
sunstone license check --json

Exits non-zero if any output has a conflict, so it slots into CI alongside sunstone lint and sunstone dataset validate.

Example output (compatible):

enriched-members: target=CC-BY-SA-4.0 status=compatible
public-summary: target=CC-BY-4.0 status=compatible

Example output (conflict):

nc-derived: target=CC-BY-4.0 status=conflict
  conflict: CC-BY-NC-4.0 is NonCommercial: derivatives must also be NonCommercial, not CC-BY-4.0
  suggestions: CC-BY-NC-4.0, CC-BY-NC-SA-4.0, CC-BY-NC-3.0-IGO

Skipped outputs:

The check is skipped (status skipped) for an output when it has no source licenses to consider, or when no effective target license can be resolved. Skipped outputs do not fail the command.

Unknown licenses:

A LicenseRef-* identifier that is not in the embedded registry is reported as an unknown source on the JSON result and excluded from rules-based comparisons — callers (or reviewers) must decide how to treat it.

Common Workflows¶

Pre-commit Validation¶

Add validation to your CI/CD pipeline:

# .github/workflows/validate.yml
- name: Validate datasets
  run: sunstone dataset validate

Lock Datasets for Production¶

Before deploying to production, lock all datasets:

sunstone dataset lock
git add datasets.yaml
git commit -m "Lock datasets for production"

Build and Push Pipeline¶

Automate package building and publishing:

#!/bin/bash
set -e

# Validate first
sunstone dataset validate

# Build package
sunstone package build

# Push to appropriate environment
ENV=${1:-dev}
sunstone package push --env $ENV

Shell Completion¶

Enable tab completion for dataset slugs:

# Bash
eval "$(_SUNSTONE_COMPLETE=bash_source sunstone)"

# Zsh
eval "$(_SUNSTONE_COMPLETE=zsh_source sunstone)"

# Fish
_SUNSTONE_COMPLETE=fish_source sunstone | source

After enabling completion, you can tab-complete dataset slugs:

sunstone dataset validate <TAB>
# Shows: school-data  teacher-data  school-summary  analysis-results