diff --git a/README.md b/README.md index c1efba8..402cd40 100644 --- a/README.md +++ b/README.md @@ -26,27 +26,37 @@ You can install it by following the [documentation](https://docs.oscar.grycap.ne ### ๐Ÿง‘โ€๐Ÿ’ป Setting Up the Configuration File -The test suite uses environment variables to store sensitive information such as endpoints and credentials. +The test suite uses environment variables to store sensitive information such as endpoints and credentials. I'd recommend that you have two environment files. The first includes the cluster information, and the second contains the authentication process credentials. This way, you can switch between authentication processes such as EGI-CheckIn or Keycloak. Also, you can create one environment file that contains all the information. Create a `.env.yaml` file according to the template shown in `env-template.yaml` -The following information is required: - +The following information is required about the cluster information: - `OSCAR_ENDPOINT`: The endpoint of the OSCAR cluster (e.g. https://mycluster.oscar.grycap.net) - `OSCAR_METRICS`: The endpoint of the OSCAR metrics. - `OSCAR_DASHBOARD`: The endpoint of the OSCAR UI (dashboard). - `BASIC_USER:`: Base64-encoded information of the authentication for the 'oscar' user (echo -n "oscar:password" | base64) - - `EGI_AAI_URL`: The base URL of the EGI AAI (Authentication and Authorisation Infrastructure) server. - - For the production server, use `https://aai.egi.eu`. - - For the demo server, use `https://aai-demo.egi.eu`. - - `REFRESH_TOKEN`: The OIDC token used to automate the execution of the test suite. In order to get a Refresh Token, head to the [Check-in Token Portal](https://aai.egi.eu/token/) or [Demo Check-in Token Portal](https://aai-demo.egi.eu/token/), click **Authorise** and then **Create Refresh Token** button to generate a new token. - - `EGI_VO`: The virtual organization used to test the OSCAR cluster. - - `FIRST_USER`: User ID - - `FIRST_USER_ID`: Get the first 10 characters of FIRST_USER (e.g. FIRST_USER: 1234567890987654321 -> FIRST_USER_ID: 1234567890) - - `REFRESH_TOKEN_SECOND_USER`: The OIDC token of the second user used to automate the execution - - `SECOND_USER`: User ID of the second user - - `SECOND_USER_ID`: Get the first 10 characters of SECOND_USER +The next parameters are required to configure the authentication process: + - `AUTHENTICATION_PROCESS`: This parameter selects the authentication process between EGI `resources/token-egi.resource` and Keycloak `resources/token-keycloak.resource`. **ALWAYS REQUIRED**. + - `AAI_URL`: The URL token of the AAI (Authentication and Authorisation Infrastructure) server. **ALWAYS REQUIRED**. + - For the EGI production server, use `https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/token`. + - For the EGI demo server, use `https://aai-demo.egi.eu/auth/realms/egi/protocol/openid-connect/token`. + - `AAI_GROUP`: The virtual organization used to test the OSCAR cluster. **ALWAYS REQUIRED**. + - `CLIENT_ID`: Client ID of Keycloak. Only needed in Keycloak. + - `SCOPE`: Scope of Keycloak. Only needed in Keycloak. + - `FIRST_USER`: User ID. **ALWAYS REQUIRED**. + - `REFRESH_TOKEN`: The OIDC token used to automate the execution of the test suite. In order to get a Refresh Token, head to the [Check-in Token Portal](https://aai.egi.eu/token/) or [Demo Check-in Token Portal](https://aai-demo.egi.eu/token/), click **Authorise** and then **Create Refresh Token** button to generate a new token. Only used in EGI. + - `KEYCLOAK_USERNAME` and `KEYCLOAK_PASSWORD`: The user/password Keycloak authentication. Only used in Keycloak. + +In case you are testing isolation or visibility, you have to add a second user: + - `SECOND_USER`: User ID of the second user. **ALWAYS REQUIRED**. + - `REFRESH_TOKEN_SECOND_USER`: The OIDC token of the second user used to automate the execution. + - `KEYCLOAK_USERNAME_AUX` and `KEYCLOAK_PASSWORD_AUX`: The user/password of a second user in Keycloak. + +In case you are testing the mount feat using an external OSCAR cluster add,: + - `OSCAR_EXTERNAL`: Endpoint of an external OSCAR cluster. + - `MINIO_EXTERNAL`: MinIO endpoint of external OSCAR cluster. + - `MINIO_SECRET_KEY`: Secret Key of `FIRST_USER` used the `MINIO_EXTERNAL`. ### ๐Ÿงช Running Tests diff --git a/resources/files.resource b/resources/files.resource index 3180790..3e1f0de 100644 --- a/resources/files.resource +++ b/resources/files.resource @@ -8,7 +8,7 @@ Library yaml *** Variables *** ${OSCAR_ENDPOINT} ${OSCAR_ENDPOINT} -${VO} ${AAI_VO} +${VO} ${AAI_GROUP} ${DATA_DIR} ${EXECDIR}/data ${INVOKE_FILE_NAME} 00-cowsay-invoke-body.json ${INVOKE_FILE} ${DATA_DIR}/${INVOKE_FILE_NAME} diff --git a/resources/token-egi.resource b/resources/token-egi.resource new file mode 100644 index 0000000..0c87dcc --- /dev/null +++ b/resources/token-egi.resource @@ -0,0 +1,78 @@ +*** Settings *** +Documentation Shared keywords and variables for handling OIDC authentication + +Library Collections +Library DateTime +Library Process +Library RequestsLibrary +Library JSONLibrary + + +*** Variables *** +${REFRESH_TOKEN} ${REFRESH_TOKEN} +${TOKEN_URL} ${AAI_URL} +${CLIENT_ID} token-portal +${SCOPE} openid%20email%20profile%20voperson_id%20eduperson_entitlement + + +*** Keywords *** +Check Valid OIDC Token + [Documentation] Get the access token + [Tags] create delete + ${token}= Get Access Token + Check JWT Expiration ${token} + +Get Access Token + [Documentation] Retrieve OIDC token using a refresh token + ${result}= Run Process curl -s -X POST '${TOKEN_URL}' -d + ... 'grant_type\=refresh_token&refresh_token\=${REFRESH_TOKEN}&client_id\=${CLIENT_ID}&scope\=${SCOPE}' + ... shell=True stdout=True stderr=True + ${json_output}= Convert String To Json ${result.stdout} + ${access_token}= Get Value From Json ${json_output} $.access_token + VAR ${access_token}= ${access_token}[0] + Log Access Token: ${access_token} + VAR &{HEADERS}= Authorization=Bearer ${access_token} Content-Type=text/json Accept=application/json + ... scope=SUITE + VAR &{HEADERS_OSCAR}= Authorization=Basic ${BASIC_USER} Content-Type=text/json Accept=application/json + ... scope=SUITE + RETURN ${access_token} + +Decode JWT Token + [Documentation] Decode a JWT token and returns its payload + [Arguments] ${token} + ${decoded}= Evaluate + ... jwt.decode('${token}', options={"verify_signature": False}, algorithms=["HS256", "RS256"]) + RETURN ${decoded} + +Check JWT Expiration + [Documentation] Check if the given JWT token is expired + [Arguments] ${token} + ${decoded_token}= Decode JWT Token ${token} + Log ${decoded_token} + ${expiry_time}= Get From Dictionary ${decoded_token} exp + Log Token Expiration Time: ${expiry_time} + ${current_time}= Get Current Date result_format=epoch + Log Current Time: ${current_time} + Should Be True ${expiry_time} > ${current_time} Token is expired + + +Checks Valids OIDC Token + [Documentation] Get the access token + ${result}= Run Process curl -s -X POST '${TOKEN_URL}' -d + ... 'grant_type\=refresh_token&refresh_token\=${REFRESH_TOKEN}&client_id\=${CLIENT_ID}&scope\=${SCOPE}' + ... shell=True stdout=True stderr=True + ${json_output}= Convert String To Json ${result.stdout} + ${token}= Get Value From Json ${json_output} $.access_token + VAR ${token}= ${token}[0] + Check JWT Expiration ${token} + VAR &{HEADERS}= Authorization=Bearer ${token} Content-Type=text/json Accept=application/json + ... scope=SUITE + ${result}= Run Process curl -s -X POST '${TOKEN_URL}' -d + ... 'grant_type\=refresh_token&refresh_token\=${REFRESH_TOKEN_SECOND_USER}&client_id\=${CLIENT_ID}&scope\=${SCOPE}' + ... shell=True stdout=True stderr=True + ${json_output}= Convert String To Json ${result.stdout} + ${token2}= Get Value From Json ${json_output} $.access_token + VAR ${token2}= ${token2}[0] + Check JWT Expiration ${token2} + VAR &{HEADERS2}= Authorization=Bearer ${token2} Content-Type=text/json Accept=application/json + ... scope=SUITE diff --git a/tests/oscar_mount_external.robot b/tests/oscar_mount_external.robot deleted file mode 100644 index fc8bcb2..0000000 --- a/tests/oscar_mount_external.robot +++ /dev/null @@ -1,195 +0,0 @@ -*** Settings *** -Documentation Tests for the OSCAR Manager's API of a deployed OSCAR cluster. Basic endpoint coverage - -Resource ${CURDIR}/../../${RESOURCE_TO_USE} -Resource ${CURDIR}/../resources/files.resource - -Suite Setup Check Valid OIDC Token -Suite Teardown Clean Test Artifacts True ${DATA_DIR}/service_file.json - - -*** Variables *** -${SERVICE_NAME} robot-test-cowsay -${BUCKET_EXTERNAL} mount-external-bucket - -*** Test Cases *** -OSCAR API Health - [Documentation] Check API health - ${response}= GET ${OSCAR_ENDPOINT}/health expected_status=200 - Log ${response.content} - Should Be Equal As Strings ${response.content} Ok - -OSCAR API Health of external cluster - [Documentation] Check API health - ${response}= GET ${OSCAR_EXTERNAL}/health expected_status=200 - Log ${response.content} - Should Be Equal As Strings ${response.content} Ok - -OSCAR CLI Cluster Add - [Documentation] Check that OSCAR CLI adds a cluster - [Tags] create delete - ${result}= Run Process oscar-cli cluster add robot-oscar-cluster ${OSCAR_ENDPOINT} - ... --oidc-refresh-token ${REFRESH_TOKEN} stdout=True stderr=True - Log ${result.stdout} - # Should Be Equal As Integers ${result.rc} 0 - Should Contain ${result.stdout} successfully - - - -OSCAR CLI Cluster Default - [Documentation] Check that OSCAR CLI sets a cluster as default - [Tags] create delete - ${result}= Run Process oscar-cli cluster default --set robot-oscar-cluster - ... stdout=True stderr=True - Log ${result.stdout} - # Should Be Equal As Integers ${result.rc} 0 - Should Contain ${result.stdout} successfully - - -Verify Bucket Private creation - [Documentation] List all buckets and check is private - ${response}= External Verify Bucket - Should Be Equal As Strings ${response.status_code} 200 - ${output} = Convert To String ${response.content} - IF '"bucket_name":"${BUCKET_EXTERNAL}","visibility":"private"' not in '${output}' - Log "Not bucket, let's create" - ${body}= Get File ${DATA_DIR}/bucket.json - ${body}= yaml.Safe Load ${body} - Set To Dictionary ${body} bucket_name=${BUCKET_EXTERNAL} - ${body}= Convert JSON To String ${body} - Log ${body} - ${response}= POST url=${OSCAR_EXTERNAL}/system/buckets expected_status=201 data=${body} - ... headers=${HEADERS} - Should Be Equal As Strings ${response.status_code} 201 - ${response}= External Verify Bucket - Should Be Equal As Strings ${response.status_code} 200 - Log ${response.content} - Should Contain ${response.content} "bucket_name":"${BUCKET_EXTERNAL}","visibility":"private" - END - - - -OSCAR Create Service Mount - where the bucket ${BUCKET_EXTERNAL} exist and its mine and private - Prepare Service File - ${body}= Get File ${DATA_DIR}/service_file.json - ${body}= yaml.Safe Load ${body} - ${mount} = Create Dictionary storage_provider=minio.external path=${BUCKET_EXTERNAL} - ${external} = Create Dictionary endpoint=${MINIO_EXTERNAL} access_key=${MINIO_ACCESS_KEY} - ... secret_key=${MINIO_SECRET_KEY} verify=${True} region=us-east-1 - ${storage} = Create Dictionary - ${minio} = Create Dictionary - Set To Dictionary ${body} mount=${mount} - Set To Dictionary ${storage} external=${external} - Set To Dictionary ${minio} minio=${storage} - Set To Dictionary ${body} storage_providers=${minio} - ${body}= Convert JSON To String ${body} - Log ${body} - ${response}= POST url=${OSCAR_ENDPOINT}/system/services data=${body} - ... headers=${HEADERS} - Log ${response.content} - Log ${response} - Should Be Equal As Strings ${response.status_code} 201 - - -OSCAR CLI Put File to external bucket - [Documentation] Check that OSCAR CLI puts a file in a service's storage provider - ${result}= Run Process oscar-cli service put-file ${SERVICE_NAME} minio.external - ... ${EXECDIR}/data/00-cowsay-invoke-body.json ${BUCKET_EXTERNAL}/${INVOKE_FILE_NAME} - ... stdout=True stderr=True - Log ${result.stdout} - Should Be Equal As Integers ${result.rc} 0 - - -OSCAR CLI List Files - [Documentation] Check that OSCAR CLI lists files from a service's storage provider path - ${result}= Run Process oscar-cli service list-files ${SERVICE_NAME} - ... minio.external ${BUCKET_EXTERNAL} - Log ${result.stdout} - # Should Be Equal As Integers ${result.rc} 0 - Should Contain ${result.stdout} ${INVOKE_FILE_NAME} - - -OSCAR CLI Put File - [Documentation] Check that OSCAR CLI puts a file in a service's storage provider - ${result}= Run Process oscar-cli service put-file ${SERVICE_NAME} minio - ... ${EXECDIR}/data/00-cowsay-invoke-body.json robot-test/input/${INVOKE_FILE_NAME} - ... stdout=True stderr=True - Log ${result.stdout} - Should Be Equal As Integers ${result.rc} 0 - Sleep 20s - - -OSCAR List Jobs - [Documentation] List all jobs from a service with their status - ${list_jobs}= GET url=${OSCAR_ENDPOINT}/system/logs/${SERVICE_NAME} expected_status=200 - ... headers=${HEADERS} - ${jobs_dict}= Evaluate dict(${list_jobs.content}) - Get Key From Dictionary ${jobs_dict} - Should Contain ${JOB_NAME} ${SERVICE_NAME}- - Sleep 20s - -OSCAR Get Logs - [Documentation] Get the logs from a job - ${get_logs}= GET url=${OSCAR_ENDPOINT}/system/logs/${SERVICE_NAME}/${JOB_NAME} expected_status=200 - ... headers=${HEADERS} - Log ${get_logs.content} - Should Contain ${get_logs.content} Hello - - -OSCAR Delete Service ${SERVICE_NAME} - [Documentation] Delete the created service - [Tags] delete - ${response}= DELETE url=${OSCAR_ENDPOINT}/system/services/${SERVICE_NAME} expected_status=204 - ... headers=${HEADERS} - Log ${response.content} - Should Be Equal As Strings ${response.status_code} 204 - ${response}= GET url=${OSCAR_ENDPOINT}/system/services/${SERVICE_NAME} expected_status=404 - ... headers=${HEADERS} - - -Delete Bucket Private - [Documentation] Delete a public bucket - [Tags] Delete bucket - ${response}= DELETE url=${OSCAR_EXTERNAL}/system/buckets/${BUCKET_EXTERNAL} expected_status=204 - ... headers=${HEADERS} - Log ${response.content} - Should Be Equal As Strings ${response.status_code} 204 - -Verify Bucket Public Delete - [Documentation] List all buckets 3 - ${response}= External Verify Bucket - Should Be Equal As Strings ${response.status_code} 200 - ${output} = Convert To String ${response.status_code} - Should Not Match Regexp ${output} ${bucket_name} - - - - - -*** Keywords *** -Prepare Service File - [Documentation] Prepare the service file - ${service_content}= Get File ${DATA_DIR}/00-cowsay.yaml - ${service_content}= Set Service File VO ${service_content} - - # Extract the inner dictionary (remove 'functions', 'oscar' and 'robot-oscar-cluster') - VAR ${modified_content}= ${service_content}[functions][oscar][0][robot-oscar-cluster] - - # Update the script value - ${script_value}= Catenate - ... \#!/bin/sh\n\nsleep 5\nif [ \"$INPUT_TYPE\" = \"json\" ]\nthen\n - ... jq '.message' \"$INPUT_FILE_PATH\" -r | /usr/games/cowsay\nelse\n - ... cat \"/mnt/${BUCKET_EXTERNAL}/${INVOKE_FILE_NAME}\" | /usr/games/cowsay\nfi\n\ - Set To Dictionary ${modified_content} script=${script_value} - ${mount} = Create Dictionary storage_provider=minio.default path=${BUCKET_EXTERNAL} - Set To Dictionary ${modified_content} mount=${mount} - - Set To Dictionary ${modified_content} script=${script_value} - ${service_content_json}= Evaluate json.dumps(${modified_content}) json - Create File ${DATA_DIR}/service_file.json ${service_content_json} - - -External Verify Bucket - [Documentation] List all buckets - ${response}= GET url=${OSCAR_EXTERNAL}/system/buckets expected_status=200 headers=${HEADERS} - RETURN ${response} \ No newline at end of file diff --git a/variables/.env-template.yaml b/variables/.env-template.yaml index 06cb412..c7db5e8 100644 --- a/variables/.env-template.yaml +++ b/variables/.env-template.yaml @@ -1,25 +1,53 @@ -# OSCAR SERVICES +# MAIN OSCAR configuration OSCAR_ENDPOINT: OSCAR_METRICS: OSCAR_DASHBOARD: +# Basic auth for OSCAR BASIC_USER: +# Other configuration +LOCAL_TESTING: False +SSL_VERIFY: True -# EGI INFO -EGI_AAI_URL: -REFRESH_TOKEN: -EGI_VO: -EGI_UID: -LOCAL_TESTING: False -SSL_VERIFY: True -FIRST_USER: -FIRST_USER_ID: +##### EGI Authentication file example +AUTHENTICATION_PROCESS: resources/token-egi.resource +AAI_URL: https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/token +AAI_VO: + +#First user +REFRESH_TOKEN: <> +FIRST_USER: <>@egi.eu #Second user for testing REFRESH_TOKEN_SECOND_USER: -SECOND_USER: -SECOND_USER_ID: +SECOND_USER: <>@egi.eu -# BASIC AUTH for OSCAR -BASIC_USER: +# For testing mounting external buckets +OSCAR_EXTERNAL: https:// +MINIO_EXTERNAL: https://minio. +MINIO_SECRET_KEY: + + + +##### Keycloak Authentication file example +AUTHENTICATION_PROCESS: resources/token-keycloak.resource +AAI_URL: https:///realms//protocol/openid-connect/token +AAI_VO: +CLIENT_ID: +SCOPE: + +#First user +KEYCLOAK_USERNAME: +KEYCLOAK_PASSWORD: +FIRST_USER: <> + +#Second user for testing +KEYCLOAK_USERNAME_AUX: +KEYCLOAK_PASSWORD_AUX: +SECOND_USER: <> + +# For testing mounting external buckets +OSCAR_EXTERNAL: https:// +MINIO_EXTERNAL: https://minio. +MINIO_SECRET_KEY: <>