Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .devin/wiki.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"repo_notes": [
{
"author": "OCM Test Suite maintainers",
"content": "This repository is the Open Cloud Mesh (OCM) interoperability test harness. It runs real browser flows against real containerized platforms, captures structured evidence, aggregates results across a matrix, and publishes them to the public OCM Observatory. It is not the OCM specification (that is cs3org/OCM-API), it is not the Observatory website (that is MahdiBaghbani/ocm-web-site), and it does not build the platform images (those come from MahdiBaghbani/containers, published to ghcr.io/mahdibaghbani/containers)."
},
{
"author": "OCM Test Suite maintainers",
"content": "The system is driven by a Nushell CLI called ocmts over a config-first design. Core vocabulary: flow_id, matrix_key, and cell_id (see docs/architecture/tuple-identity.md). A run resolves a tuple, generates Docker Compose overlays, runs a Cypress flow (with MITM capture where relevant), and writes a structured evidence envelope (meta/suite-manifest.v1.json). CI is generated from config with drift checks, aggregates results, and deploys the Observatory. Enabled flows today: login, share-with, contact-token, contact-wayf. Platforms: Nextcloud, oCIS, OpenCloud, CERNBox, OpenCloudMesh Go. Around 50 CI cells, Chrome-first."
},
{
"author": "OCM Test Suite maintainers",
"content": "Keep scope honest. This suite proves reviewable, evidence-backed interoperability for the flows and platform pairs in the matrix. It is not an official conformance or certification authority, it does not cover every platform in every flow, and it is Chrome-first rather than multi-browser. Coverage is tracked explicitly in config/adapters/capabilities.v1.nuon with states like supported, test-implementation-pending, and vendor-unsupported. Treat docs/architecture/evidence-standard.md as the contract for what a run proves."
},
{
"author": "OCM Test Suite maintainers",
"content": "History and boundaries. The project started in 2023 as an earlier OCM and EFSS interoperability lab and was deliberately rebuilt in 2026 into the current ocmts harness. It was founded by Michiel de Jong, has major historical CERNBox and Reva contributions from Giuseppe Lo Presti, and is built into its current form primarily by Mahdi Baghbani. When summarizing, emphasize the CLI, config SSOT under config/, the matrix and capability model, the evidence envelope, generated CI, and the site publish pipeline into ocm-web-site. Do not describe generated CI or generated Cypress matrix files as hand-edited."
}
]
}
92 changes: 92 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Contributing

Thanks for taking a look at the OCM Test Suite. Whether you want to fix a flaky
flow, add a platform, tighten the evidence pipeline, or improve the docs, your
help is welcome here.

This repo is config-first and automation-heavy, so the most useful thing this
guide can do is show you where things live and how not to fight the machinery.

## The mental model

- This repo runs interoperability tests and turns them into evidence.
- It does not build the platform images (those come from
[MahdiBaghbani/containers](https://github.com/MahdiBaghbani/containers), the
DockyPody image fleet).
- It does not host the Observatory UI (that is
[MahdiBaghbani/ocm-web-site](https://github.com/MahdiBaghbani/ocm-web-site));
this repo builds and deploys it from CI.
- It is not the OCM spec ([cs3org/OCM-API](https://github.com/cs3org/OCM-API)).

## Local setup

Prereqs:

- Nushell (`nu`)
- Docker (and Docker Compose)
- Bun (used for the TypeScript sidecars and CI preflight)

Start here:

```sh
nu scripts/ocmts.nu
nu scripts/ocmts.nu test units
```

`test units` runs fast, Docker-free checks of the automation logic and is the
quickest way to know you did not break the CLI.

## Config-first changes

Most changes are configuration, not hand-written glue. A few common cases:

- Change matrix rules or platforms: edit `config/matrix/*`, then regenerate
derived artifacts (Cypress matrix and CI workflows) and run the drift checks.
- Bump an image: edit `config/images.nuon` (the images themselves live in the
containers repo).
- Add or change an adapter: update the Cypress adapter registry and the
capability registry in `config/adapters/`.
- Change actors or credentials: edit `config/actors/`.

Generated files (CI workflows, workflow assets, generated Cypress matrix files)
should not be hand-edited. Regenerate them instead. See
[config/ci/README.md](config/ci/README.md) and
[scripts/README.md](scripts/README.md) for the exact commands.

## Before you open a pull request

Run the checks that CI will run:

- `nu scripts/ocmts.nu test units`
- the workflow generation and drift checks (see `config/ci/README.md`)
- the capability check for matrix changes

If your change touches a flow end to end, run at least one relevant cell locally
and confirm the evidence looks right.

## Evidence rules

Evidence is the product here, so keep it trustworthy. See
[docs/architecture/evidence-standard.md](docs/architecture/evidence-standard.md)
for the contract, and the Cypress policies (no `allowCypressEnv`, isolation
rules, IdP session handling) documented under
[docs/operations/configuration.md](docs/operations/configuration.md).

## Pull requests

- Keep pull requests focused.
- Say whether the change touches a flow, the matrix, the CI generation, or the
evidence pipeline.
- Call out cross-repo implications (images in `containers`, UI in
`ocm-web-site`).
- Update the docs when behavior changes.

## Questions and issues

Questions, bug reports, and ideas are welcome on the
[issue tracker](https://github.com/cs3org/ocm-test-suite/issues). If you are
planning a larger change, opening an issue first to talk it through saves
everyone time.

By contributing, you agree that your contributions are licensed under
AGPL-3.0-or-later, consistent with this repository.
63 changes: 63 additions & 0 deletions FUNDING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
<!--
# SPDX-License-Identifier: AGPL-3.0-or-later
# OCM Test Suite: interoperability testing and evidence pipeline
# Copyright (C) 2026 Mahdi Baghbani <mahdi-baghbani@azadehafzar.io>
#-->

# Funding

The OCM Test Suite is built and maintained by Mahdi Baghbani as part of ongoing
work on Open Cloud Mesh (OCM). Turning interoperability from a claim into
reviewable evidence, across real platforms and many versions, takes sustained
time. That time is only possible because a few organizations chose to put real
money into open source infrastructure, and I am grateful to them.

## NLnet

NLnet backed this work through the NGI0 Core Fund, a fund established by NLnet
with financial support from the European Commission's Next Generation Internet
programme, under the aegis of DG Communications Networks, Content and Technology
under grant agreement No 101092990.

<p>
<a href="https://nlnet.nl/project/OpenCloudMesh/">
<img alt="NLnet Foundation" src="assets/logos/funders/nlnet.svg" height="64">
</a>
&nbsp;&nbsp;&nbsp;
<a href="https://www.nlnet.nl/core">
<img alt="NGI0 Core" src="assets/logos/funders/ngi0-core.svg" height="64">
</a>
</p>

- Project page: <https://nlnet.nl/project/OpenCloudMesh/>
- NGI0 Core Fund: <https://www.nlnet.nl/core>
- Grant agreement 101092990: <https://cordis.europa.eu/project/id/101092990>
- NLnet: <https://nlnet.nl/>
- Next Generation Internet: <https://ngi.eu/>

## Sovereign Tech Agency

The Sovereign Tech Agency backs this work through the Sovereign Tech Fund, and
that support is a big part of what keeps the interoperability testing and the
public Observatory moving.

<p>
<a href="https://www.sovereign.tech/tech/open-cloud-mesh">
<img alt="Sovereign Tech Agency" src="assets/logos/funders/sovereign-tech-agency.svg" height="64">
</a>
&nbsp;&nbsp;&nbsp;
<a href="https://www.sovereign.tech/programs/fund">
<img alt="Sovereign Tech Fund" src="assets/logos/funders/sovereign-tech-fund.svg" height="64">
</a>
</p>

- Project page: <https://www.sovereign.tech/tech/open-cloud-mesh>
- Sovereign Tech Fund: <https://www.sovereign.tech/programs/fund>
- Sovereign Tech Agency: <https://www.sovereign.tech/>

## The wider project

For the bigger picture of Open Cloud Mesh and the protocol-level funding
credits, see the specification repository:

- <https://github.com/cs3org/OCM-API>
178 changes: 156 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,65 @@
# OCM Test Suite

End-to-end tests for Open Cloud Mesh (OCM) interoperability across multiple
providers (for example Nextcloud, oCIS, OpenCloud). The suite runs real UI
flows, captures evidence (screenshots, videos, logs, metadata), and can publish
results to a static site.
> Proof, not promises: end-to-end Open Cloud Mesh interoperability tests across
> real platforms, with the screenshots, logs, and traffic to back every result.

If you are here to run the suite, start with "Quick start". If you are here to
understand how it is built, use the docs index under `docs/`.
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/cs3org/ocm-test-suite)
[![License: AGPL-3.0](https://img.shields.io/badge/license-AGPL--3.0-blue.svg)](LICENSE.md)

Interoperability is easy to claim and hard to show. This repo exists to show it.
It stands up real Open Cloud Mesh (OCM) platforms in Docker, drives real
browser flows between them, and captures what actually happened as reviewable
evidence. Then it publishes the whole thing as a browsable compatibility matrix.

- Live results: <https://cs3org.github.io/ocm-test-suite/>
- Observatory: <https://cs3org.github.io/ocm-test-suite/observatory/>

This is the engine behind the OCM Observatory. It is not the OCM specification
(that lives in [cs3org/OCM-API](https://github.com/cs3org/OCM-API)), and it is
not the Observatory website (that lives in
[MahdiBaghbani/ocm-web-site](https://github.com/MahdiBaghbani/ocm-web-site)).
What it owns is the hard part in the middle: running the tests and turning them
into evidence anyone can inspect.

## Why this matters

When someone says "platform A and platform B interoperate over OCM," what does
that actually mean? Which flow? Which versions? And can you see it, or do you
have to take it on faith?

This suite answers those questions with artifacts. Every run captures
screenshots, video, container logs, run metadata, and, for the protocol-heavy
flows, the actual OCM traffic through a MITM proxy. A red or green box is never
the end of the story here; you can open a cell and see exactly what happened.

## What it tests today

Real UI flows against real containerized stacks, not mocked APIs:

- Platforms: Nextcloud, oCIS, OpenCloud, CERNBox, and OpenCloudMesh Go
- Flows: `login`, `share-with`, `contact-token`, `contact-wayf`
- Around 50 matrix cells run in CI, Chrome-first

Coverage is deliberately explicit. Not every platform supports every flow, and
the suite tracks that honestly with a capability registry rather than hiding
gaps. What is supported, what is pending, and what a vendor does not support are
all first-class states.

## How it works

The suite is driven by a Nushell CLI (`ocmts`) over a config-first design:

1. You pick a tuple: a flow, a sender platform and version, and (for two-party
flows) a receiver.
2. `ocmts` resolves the matrix cell, generates Docker Compose overlays, and
brings the stacks up.
3. It runs the Cypress flow in a container, wiring MITM capture where relevant.
4. It collects artifacts into a structured evidence envelope, then tears down.
5. In CI, results are aggregated across the matrix and published to the
Observatory.

The CI itself is generated from config, with drift checks, so the matrix stays
consistent instead of drifting into hand-maintained YAML.

## Quick start

Expand All @@ -15,32 +68,113 @@ Prereqs:
- Nushell (`nu`)
- Docker (and Docker Compose)

Get help and discover commands:
Discover commands:

- `nu scripts/ocmts.nu`
- `nu scripts/ocmts.nu services up run --help`
- `nu scripts/ocmts.nu test cypress suite --help`
```sh
nu scripts/ocmts.nu
nu scripts/ocmts.nu services up run --help
nu scripts/ocmts.nu test cypress suite --help
```

Run a single cell locally (brings up services, runs Cypress, collects
artifacts, then tears down unless `--keep-up` is used):
artifacts, then tears down unless `--keep-up`):

- `nu scripts/ocmts.nu services up run ...`
```sh
nu scripts/ocmts.nu services up run ...
```

Run the full enabled suite locally:

- `nu scripts/ocmts.nu test cypress suite ...`
```sh
nu scripts/ocmts.nu test cypress suite ...
```

Run fast internal unit tests (no Docker):

- `nu scripts/ocmts.nu test units`
```sh
nu scripts/ocmts.nu test units
```

Heads up: the full suite pulls multi-container images and is resource-heavy.
For one-cell runs and the exact flags, see `docs/operations/cli.md`.

## Where to read next

- **Docs index**: `docs/README.md`
- **CLI and local run details**: `docs/operations/cli.md`
- **Configuration (images, actors, Cypress env)**: `docs/operations/configuration.md`
- **Flow identity (tuple + matrix_key)**: `docs/architecture/tuple-identity.md`
- **Evidence and publication**: `docs/architecture/evidence-standard.md`,
`docs/operations/site-publish.md`
- **Automation layout**: `scripts/README.md`
- **CI workflow generation**: `config/ci/README.md`
- Docs index: [docs/README.md](docs/README.md)
- CLI and local runs: [docs/operations/cli.md](docs/operations/cli.md)
- Configuration (images, actors, Cypress env):
[docs/operations/configuration.md](docs/operations/configuration.md)
- Flow identity (tuple + matrix_key):
[docs/architecture/tuple-identity.md](docs/architecture/tuple-identity.md)
- Evidence and publication:
[docs/architecture/evidence-standard.md](docs/architecture/evidence-standard.md),
[docs/operations/site-publish.md](docs/operations/site-publish.md)
- Automation layout: [scripts/README.md](scripts/README.md)
- CI workflow generation: [config/ci/README.md](config/ci/README.md)

## DeepWiki

If you want a browsable, AI-generated map of this repository, see
[DeepWiki](https://deepwiki.com/cs3org/ocm-test-suite). It is a fast way to get
oriented across the CLI, matrix, and evidence pipeline, but the files under
[`docs/`](docs/) are still the source of truth.

## Ecosystem

This suite sits in the middle of the OCM stack:

- Spec under test: [cs3org/OCM-API](https://github.com/cs3org/OCM-API)
- Platform images it runs:
[MahdiBaghbani/containers](https://github.com/MahdiBaghbani/containers)
(DockyPody), published to `ghcr.io/mahdibaghbani/containers/*`
- Results UI it feeds:
[MahdiBaghbani/ocm-web-site](https://github.com/MahdiBaghbani/ocm-web-site)
(the Observatory)
- A peer implementation in the same effort:
[MahdiBaghbani/opencloudmesh-go](https://github.com/MahdiBaghbani/opencloudmesh-go)

Under test alongside those: Nextcloud, oCIS, OpenCloud, and CERNBox.

## History and credits

This project has a long history. It started in 2023 as an earlier OCM and EFSS
interoperability lab and grew a large body of practical Docker, Reva, and
Cypress testing knowledge. In 2026 it was deliberately rebuilt into the current
`ocmts` harness: config-driven matrix rules, a tuple identity model, structured
evidence, generated CI, and the publish pipeline behind the Observatory.

- Started by Michiel de Jong, who founded the project and led its early
direction.
- Major historical CERNBox and Reva contributions from Giuseppe Lo Presti,
which brought the CERN-side stack into the picture.
- Built into its current form primarily by Mahdi Baghbani, who leads the modern
architecture and rewrite, with contributions from other OCM collaborators.

## Acknowledgements

This work exists because a few organizations chose to fund open source
interoperability. A big thank you to NLnet and the Sovereign Tech Agency for
backing the Open Cloud Mesh work behind this suite, which Mahdi Baghbani
maintains.

<p>
<a href="https://nlnet.nl/project/OpenCloudMesh/">
<img alt="NLnet Foundation" src="assets/logos/funders/nlnet.svg" height="64">
</a>
&nbsp;&nbsp;&nbsp;
<a href="https://www.sovereign.tech/tech/open-cloud-mesh">
<img alt="Sovereign Tech Agency" src="assets/logos/funders/sovereign-tech-agency.svg" height="64">
</a>
</p>

You can read the full story in [FUNDING.md](FUNDING.md).

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](./CONTRIBUTING.md) for the local
workflow, the config-first change model, and what makes review easy.

## License

Licensed under the GNU Affero General Public License v3.0 or later
(AGPL-3.0-or-later). See [LICENSE.md](LICENSE.md).
Loading