updated change log and docs

Author: TejasMorbagal

CHANGES.md

@@ -82,3 +82,18 @@
   - S3 write credentials are resolved from `S3_USER_STORAGE_KEY`/`S3_USER_STORAGE_SECRET`,
     `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`, or the boto3 default chain
     (IAM role, `~/.aws/credentials`) — no secrets in config files.
+
+- Made `osc_project` a configurable parameter on `OscDatasetStacGenerator` (default:
+  `"deep-earth-system-data-lab"`), replacing the previously hardcoded value.
+  - The project identifier is now used dynamically when setting `osc:project` on the
+    OSC extension, generating the `related` link to the project collection, and
+    resolving the project collection file path during publishing.
+
+- Publisher now creates the OSC project collection automatically when it does not yet
+  exist in the catalog repository, instead of failing or silently skipping.
+  - A minimal but valid STAC Collection is generated for the project with `self`,
+    `root`, and `parent` links.
+  - A `child` link for the new project is appended to `projects/catalog.json` so the
+    project is reachable from the catalog root.
+  - If the project collection already exists, the existing update path (appending
+    product `child` and theme `related` links) is used unchanged.

docs/configuration.md

@@ -7,10 +7,55 @@ collection_id: your-collection
 osc_themes: [cryosphere]
 osc_region: global
 dataset_status: completed   # or ongoing/planned
+license_type: CC-BY-4.0
 documentation_link: https://example.com/docs
 access_link: s3://bucket/your-dataset.zarr
+
+# Optional: publish a STAC Catalog + Item next to the data on S3.
+# When set, a lightweight STAC hierarchy (catalog.json → item.json) is written
+# directly to S3 and a "via" link is added to the OSC collection pointing to it.
+stac_catalog_s3_root: s3://bucket/stac/your-collection/
+```
+
+### Field reference
+
+| Field | Required | Description |
+|---|---|---|
+| `dataset_id` | Yes | Zarr store identifier (used to open the dataset). |
+| `collection_id` | Yes | Unique ID for the STAC collection in the OSC catalog. |
+| `license_type` | Yes | SPDX license identifier (e.g. `CC-BY-4.0`). |
+| `osc_themes` | No | List of OSC theme slugs (e.g. `[cryosphere, oceans]`). |
+| `osc_region` | No | Geographical region label (default: `Global`). |
+| `dataset_status` | No | One of `ongoing`, `completed`, or `planned` (default: `ongoing`). |
+| `access_link` | No | Public S3 URL of the Zarr store. Defaults to `s3://deep-esdl-public/{dataset_id}`. |
+| `documentation_link` | No | URL to dataset documentation. |
+| `stac_catalog_s3_root` | No | S3 root for the dataset-level STAC Catalog/Item. See [STAC Catalog on S3](#stac-catalog-on-s3). |
+
+### STAC Catalog on S3
+
+Setting `stac_catalog_s3_root` generates a two-file STAC hierarchy on S3 alongside
+the data:
+
+```
+s3://bucket/stac/your-collection/
+├── catalog.json        # STAC Catalog (root)
+└── your-collection/
+    └── item.json       # STAC Item covering the full Zarr store
 ```
 
+The item has two assets:
+
+- `zarr-data` — points to the Zarr store (`application/vnd+zarr`).
+- `zarr-consolidated-metadata` — points to `.zmetadata` (`application/json`).
+
+The OSC collection gains a `via` link to `catalog.json` so STAC-aware clients
+can discover the data path. `rel="child"` is intentionally avoided because the
+OSC validator requires every `child` link to resolve inside the metadata repository.
+
+S3 credentials are resolved in this order: `S3_USER_STORAGE_KEY` /
+`S3_USER_STORAGE_SECRET` env vars, then `AWS_ACCESS_KEY_ID` /
+`AWS_SECRET_ACCESS_KEY`, then the boto3 default chain (IAM role, `~/.aws/credentials`).
+
 ## Workflow config (YAML)
 ```yaml
 workflow_id: your-workflow

docs/python-api.md

@@ -8,7 +8,7 @@ from deep_code.tools.publish import Publisher
 publisher = Publisher(
     dataset_config_path="dataset.yaml",
     workflow_config_path="workflow.yaml",
-    environment="staging",
+    environment="staging",  # "production" | "staging" | "testing"
 )
 
 # Generate files locally (no PR)
@@ -18,3 +18,72 @@ publisher.publish(write_to_file=True, mode="all")
 publisher.publish(write_to_file=False, mode="dataset")
 ```
 
+`mode` controls what is published:
+
+| `mode` | What is published |
+|---|---|
+| `"dataset"` | OSC STAC collection, variable catalogs, product/variable base catalogs, project collection. |
+| `"workflow"` | OGC API workflow and experiment records, workflow/experiment base catalogs. |
+| `"all"` | Both of the above (default). |
+
+---
+
+## OscDatasetStacGenerator
+
+`OscDatasetStacGenerator` can also be used directly when you need more control
+over individual artifacts.
+
+```python
+from deep_code.utils.dataset_stac_generator import OscDatasetStacGenerator
+
+generator = OscDatasetStacGenerator(
+    dataset_id="my-dataset.zarr",
+    collection_id="my-collection",
+    workflow_id="my-workflow",
+    workflow_title="My Workflow",
+    license_type="CC-BY-4.0",
+    osc_themes=["cryosphere"],
+    osc_region="Global",
+    osc_status="completed",
+    # Optional: override the default project identifier.
+    # Controls osc:project on the collection and the link to the project collection.
+    osc_project="deep-earth-system-data-lab",
+)
+```
+
+### `osc_project` parameter
+
+`osc_project` defaults to `"deep-earth-system-data-lab"` and is used in three places:
+
+1. Sets `osc:project` on the OSC extension of the generated STAC collection.
+2. Generates the `related` link from the product collection to the project collection
+   (`../../projects/{osc_project}/collection.json`).
+3. Determines the file path of the project collection when publishing
+   (`projects/{osc_project}/collection.json`).
+
+### Automatic project collection creation
+
+When `Publisher.publish_dataset()` runs, it checks whether
+`projects/{osc_project}/collection.json` already exists in the catalog repository:
+
+- **Missing** — a minimal STAC Collection is created for the project and a `child`
+  link is appended to `projects/catalog.json` so it is reachable from the catalog root.
+- **Exists** — the existing collection is updated with a `child` link to the new
+  product and `related` links for its themes (same behaviour as before).
+
+This means publishing to a new project does not require manual catalog setup.
+
+### STAC Catalog and Item generation
+
+```python
+# Build the S3 STAC hierarchy (dict keyed by S3 path)
+file_dict = generator.build_zarr_stac_catalog_file_dict(
+    stac_catalog_s3_root="s3://bucket/stac/my-collection/"
+)
+# file_dict contains:
+#   "s3://bucket/stac/my-collection/catalog.json"
+#   "s3://bucket/stac/my-collection/my-collection/item.json"
+```
+
+See [STAC Catalog on S3](configuration.md#stac-catalog-on-s3) for details on the
+generated structure.