Create python tool to call validator, and connect to MinIO using Keycloak credentials

[douglowe]

Expanding on previous work, create a demonstration python script that will request the validation of a ro-crate on minio, getting keycloak credentials for this and passing them to the validator via the api.

[douglowe]

This is the previous task to build on: eScienceLab/5S-TES-deployment#56

[douglowe]

@nimpo - can you advise on how we get ephemeral credentials from the keycloak server for this task?

[EttoreM]

@douglowe @AnnieZQH

I had a look at the current demo stack and the Cratey-Validator implementation, and I think there are a couple of points to be clarified before building the demo script.

My current understanding of the intended flow is:

1. authenticate with Keycloak
2. use the resulting credentials/token to upload an RO-Crate to MinIO
3. call the validator API so it can read the RO-Crate from MinIO and validate it

If this is correct, I'd like to point out that the validation endpoint expects a minio_config object containing:

• endpoint
• accesskey
• secret
• ssl
• bucket

So my questions are:

1. Should the demo script use Keycloak credentials only for authentication, and then still use static MinIO access key / secret for upload and validation?
2. Is the validator API itself expected to be called with Keycloak authentication (which is currently not the case), or is Keycloak only relevant for MinIO access?

[douglowe]

1. The demo script should use keycloak credentials for accessing MinIO. If we need to upload the example crate separately then that can be done with the static MinIO access key / secret.
2. Keycloak should only be relevant for MinIO access. We will wont expose the validator api to the outside world, so it shouldn't need authentication at the moment.

[nimpo]

@nimpo - can you advise on how we get ephemeral credentials from the keycloak server for this task?

. o O ( thinking )

[EttoreM]

This is what I gathered so far.

The flow

The fhow should be:

1. Keycloak login returns OIDC tokens
2. MinIO STS should exchange the access_token for temporary object-store credentials.
3. Those temporary credentials are what should map to the validator’s accesskey and secret.

The problem

While Keycloak does return the token upon login, MinIO STS seems not to be working properly. In particular, when passing the token to it on http://localhost:9000, MinIO returns the following:

<Code>STSIAMNotInitialized</Code>
<Message>IAM sub-system is being initialized, please try again</Message>

suggesting that the STS/IAM subsystem is not ready to issue temporary credentials.

Possible fixes (??)

I wonder whether MinIO tries to initialise OIDC against Keycloak before Keycloak is ready. Maybe we should add something like the following to the 5S-TES-deployment/ServiceStack/compose-manifests/storage/minio-submission.yml and 5S-TES-deployment/ServiceStack/compose-manifests/storage/minio-tre.yml

depends_on:
  keycloak:
    condition: service_healthy

Things to check

Is it possible that there is some issues with the MinIO image itself rather than with the DemosStack compose wiring and/or startup ordering? For example MinIO OIDC/STS might have configuration detail that are wrong or missing.