Mig release 5.7.0 rc8

LICENSE

@@ -1,6 +1,6 @@
-The MIT License (MIT)
+MIT License
 
-Copyright (c) 2015 EkStep
+Copyright (c) 2021 Project Sunbird
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -19,4 +19,3 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
-

README.md

@@ -12,3 +12,148 @@ The platform contains of the following projects:
 * platform-modules - Functional/Pedagogy modules to support game based learning and API
 
 
+## learning-service local setup
+This readme file contains the instruction to set up and run the learnin-service in local machine.
+
+### System Requirements:
+
+### Prerequisites:
+* Java 11 is required for docker container setups
+* Java 8 and Tomcat v9 is required for local service setup via IntelliJ
+
+### Prepare folders for database data and logs
+
+```shell
+mkdir -p ~/sunbird-dbs/neo4j ~/sunbird-dbs/cassandra ~/sunbird-dbs/redis ~/sunbird-dbs/es ~/sunbird-dbs/kafka
+export sunbird_dbs_path=~/sunbird-dbs
+```
+
+
+### Elasticsearch database setup in docker:
+```shell
+docker run --name sunbird_es -d -p 9200:9200 -p 9300:9300 \
+-v $sunbird_dbs_path/es/data:/usr/share/elasticsearch/data \
+-v $sunbird_dbs_path/es/logs://usr/share/elasticsearch/logs \
+-v $sunbird_dbs_path/es/backups:/opt/elasticsearch/backup \
+ -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.8.22
+
+```
+> --name -  Name your container (avoids generic id)
+>
+> -p - Specify container ports to expose
+>
+> Using the -p option with ports 7474 and 7687 allows us to expose and listen for traffic on both the HTTP and Bolt ports. Having the HTTP port means we can connect to our database with Neo4j Browser, and the Bolt port means efficient and type-safe communication requests between other layers and the database.
+>
+> -d - This detaches the container to run in the background, meaning we can access the container separately and see into all of its processes.
+>
+> -v - The next several lines start with the -v option. These lines define volumes we want to bind in our local directory structure so we can access certain files locally.
+>
+> --env - Set config as environment variables for Neo4j database
+>
+
+
+### Neo4j database setup in docker:
+1. First, we need to get the neo4j image from docker hub using the following command.
+```shell
+docker pull neo4j:3.3.0 
+```
+2. We need to create the neo4j instance, By using the below command we can create the same and run in a container.
+```shell
+docker run --name sunbird_neo4j -p7474:7474 -p7687:7687 -d \
+    -v $sunbird_dbs_path/neo4j/data:/var/lib/neo4j/data \
+-v $sunbird_dbs_path/neo4j/logs:/var/lib/neo4j/logs \
+-v $sunbird_dbs_path/neo4j/plugins:/var/lib/neo4j/plugins \
+--env NEO4J_dbms_connector_https_advertised__address="localhost:7473" \
+--env NEO4J_dbms_connector_http_advertised__address="localhost:7474" \
+--env NEO4J_dbms_connector_bolt_advertised__address="localhost:7687" \
+--env NEO4J_AUTH=none \
+neo4j:3.3.0
+```
+> --name -  Name your container (avoids generic id)
+>
+> -p - Specify container ports to expose
+>
+> Using the -p option with ports 7474 and 7687 allows us to expose and listen for traffic on both the HTTP and Bolt ports. Having the HTTP port means we can connect to our database with Neo4j Browser, and the Bolt port means efficient and type-safe communication requests between other layers and the database.
+>
+> -d - This detaches the container to run in the background, meaning we can access the container separately and see into all of its processes.
+>
+> -v - The next several lines start with the -v option. These lines define volumes we want to bind in our local directory structure so we can access certain files locally.
+>
+> --env - Set config as environment variables for Neo4j database
+>
+> Using Docker on Windows will also need a couple of additional configurations because the default 0.0.0.0 address that is resolved with the above command does not translate to localhost in Windows. We need to add environment variables to our command above to set the advertised addresses.
+>
+> By default, Neo4j requires authentication and requires us to first login with neo4j/neo4j and set a new password. We will skip this password reset by initializing the authentication none when we create the Docker container using the --env NEO4J_AUTH=none.
+
+3. Load seed data to neo4j using the instructions provided in the [link](master-data/loading-seed-data.md#loading-seed-data-to-neo4j-database)
+
+4. Verify whether neo4j is running or not by accessing neo4j browser(http://localhost:7474/browser).
+
+5. To SSH to neo4j docker container, run the below command.
+```shell
+docker exec -it sunbird_neo4j bash
+```
+
+### Redis database setup in docker:
+1. we need to get the redis image from docker hub using the below command.
+```shell
+docker pull redis:6.0.8 
+```
+2. We need to create the redis instance, By using the below command we can create the same and run in a container.
+```shell
+docker run --name sunbird_redis -d -p 6379:6379 redis:6.0.8
+```
+3. To SSH to redis docker container, run the below command
+```shell
+docker exec -it sunbird_redis bash
+```
+### cassandra database setup in docker:
+1. we need to get the cassandra image and can be done using the below command.
+```shell
+docker pull cassandra:3.11.8 
+```
+2. We need to create the cassandra instance, By using the below command we can create the same and run in a container.
+```shell
+docker run --name sunbird_cassandra -d -p 9042:9042 \
+-v $sunbird_dbs_path/cassandra/data:/var/lib/cassandra \
+-v $sunbird_dbs_path/cassandra/logs:/opt/cassandra/logs \
+-v $sunbird_dbs_path/cassandra/backups:/mnt/backups \
+--network bridge cassandra:3.11.8 
+```
+For network, we can use the existing network or create a new network using the following command and use it.
+```shell
+docker network create sunbird_db_network
+```
+3. To start cassandra cypher shell run the below command.
+```shell
+docker exec -it sunbird_cassandra cqlsh
+```
+4. To ssh to cassandra docker container, run the below command.
+```shell
+docker exec -it sunbird_cassandra /bin/bash
+```
+5. Load seed data to cassandra using the instructions provided in the [link](master-data/loading-seed-data.md#loading-seed-data-to-cassandra-database)
+
+
+### Steps to start learning-service in debug or development mode using IntelliJ:
+1. Navigate to downloaded repository folder and run below command in terminal.
+```shell
+export JAVA_HOME = #JDK1.8 folder location
+mvn clean install -DskipTests
+``` 
+2. Open the project in IntelliJ.
+3. Add 'Smart Tomcat' plugin. (File -> Settings -> plugin -> Search for 'Smart Tomcat' -> install)
+4. Configure the tomcat server using 'Smart Tomcat' plugin under 'Add configuration'.
+![img.png](img.png)
+5. Give the 'name' as 'learning-service'. 
+6. Specify the Tomcat v9 server location in 'Tomcat Server'. 
+7. Specify the absolute path of the 'webapp' folder (under ../sunbird-learning-platform/platform-modules/service/src/main/webapp).
+8. Mention '/learning-service' as the context path. Click on 'Apply'.
+9. Click on 'File' -> 'Project Structure'. Check if Java version is set to jdk1.8 in both 'Project' and 'Modules'.
+![img_1.png](img_1.png)
+![img_2.png](img_2.png)
+10. Click on 'File' -> Settings -> Build, Execution, Deployment -> compiler -> scala compiler -> scala compiler server. Check if Java version is set to jdk1.8.
+![img_3.png](img_3.png)
+11. Service configuration file is available at '../sunbird-learning-platform/platform-modules/service/src/main/resources/application.conf'. Update config file to connect to local databases.
+12. Start Redis, neo4j, cassandra and ElasticSearch docker containers.
+13. Start the 'learning-service' configuration in IntelliJ. Verify the health of learning service by trying to connect to 'http://localhost:8080/learning-service/health'.

ansible/artifacts-download.yml

@@ -3,8 +3,53 @@
   become: yes
   vars_files:
     - "{{inventory_dir}}/secrets.yml"
-  environment:
-    AZURE_STORAGE_ACCOUNT: "{{sunbird_artifact_storage_account_name}}"
-    AZURE_STORAGE_SAS_TOKEN: "{{sunbird_artifact_storage_account_sas}}"
-  roles:
-    - artifacts-download-azure
+  tasks:
+    - name: download artifact from azure storage
+      include_role:
+        name: azure-cloud-storage
+        tasks_from: blob-download.yml
+      vars:
+        blob_container_name: "{{ cloud_storage_artifacts_bucketname }}"
+        blob_file_name: "{{ artifact }}"
+        local_file_or_folder_path: "{{ artifact_path }}"
+        storage_account_name: "{{ cloud_artifact_storage_accountname }}"
+        storage_account_key: "{{ cloud_artifact_storage_secret }}"
+      when: cloud_service_provider == "azure"
+
+    - name: download artifact from gcloud storage
+      include_role:
+        name: gcp-cloud-storage
+        tasks_from: download.yml
+      vars:
+        gcp_storage_service_account_name: "{{ cloud_artifact_storage_accountname }}"
+        gcp_storage_key_file: "{{ cloud_artifact_storage_secret }}"
+        gcp_bucket_name: "{{ cloud_storage_artifacts_bucketname }}"
+        gcp_path: "{{ artifact }}"
+        local_file_or_folder_path: "{{ artifact_path }}"
+      when: cloud_service_provider == "gcloud"
+
+    - name: download artifact from aws s3
+      include_role:
+        name: aws-cloud-storage
+        tasks_from: download.yml
+      vars:
+        local_file_or_folder_path: "{{ artifact_path }}"
+        s3_bucket_name: "{{ cloud_storage_artifacts_bucketname }}"
+        s3_path: "{{ artifact }}"
+        aws_default_region: "{{ cloud_public_storage_region }}"
+        aws_access_key_id: "{{ cloud_artifact_storage_accountname }}"
+        aws_secret_access_key: "{{ cloud_artifact_storage_secret }}"
+      when: cloud_service_provider == "aws"
+
+    - name: download artifact from oci oss storage
+      include_role:
+        name: oci-cloud-storage
+        apply:
+          environment:
+           OCI_CLI_AUTH: "instance_principal"        
+        tasks_from: download.yml
+      vars:
+        oss_bucket_name: "{{ cloud_storage_artifacts_bucketname }}"
+        oss_object_name: "{{ artifact }}"
+        local_file_or_folder_path: "{{ artifact_path }}"
+      when: cloud_service_provider == "oci"

ansible/artifacts-upload.yml

@@ -3,8 +3,55 @@
   become: yes
   vars_files:
     - "{{inventory_dir}}/secrets.yml"
-  environment:
-    AZURE_STORAGE_ACCOUNT: "{{sunbird_artifact_storage_account_name}}"
-    AZURE_STORAGE_SAS_TOKEN: "{{sunbird_artifact_storage_account_sas}}"
-  roles:
-    - artifacts-upload-azure
+  tasks:
+    - name: upload artifact to azure storage
+      include_role:
+        name: azure-cloud-storage
+        tasks_from: blob-upload.yml
+      vars:
+        blob_container_name: "{{ cloud_storage_artifacts_bucketname }}"
+        container_public_access: "off"
+        blob_file_name: "{{ artifact }}"
+        local_file_or_folder_path: "{{ artifact_path }}"
+        storage_account_name: "{{ cloud_artifact_storage_accountname }}"
+        storage_account_key: "{{ cloud_artifact_storage_secret }}"
+      when: cloud_service_provider == "azure"
+
+    - name: upload artifact to gcloud storage
+      include_role:
+        name: gcp-cloud-storage
+        tasks_from: upload.yml
+      vars:
+        gcp_storage_service_account_name: "{{ cloud_artifact_storage_accountname }}"
+        gcp_storage_key_file: "{{ cloud_artifact_storage_secret }}"
+        gcp_bucket_name: "{{ cloud_storage_artifacts_bucketname }}"
+        gcp_path: "{{ artifact }}"
+        local_file_or_folder_path: "{{ artifact_path }}"
+      when: cloud_service_provider == "gcloud"
+
+
+    - name: upload artifact to aws s3
+      include_role:
+        name: aws-cloud-storage
+        tasks_from: upload.yml
+      vars:
+        local_file_or_folder_path: "{{ artifact_path }}"
+        s3_bucket_name: "{{ cloud_storage_artifacts_bucketname }}"
+        s3_path: "{{ artifact }}"
+        aws_default_region: "{{ cloud_public_storage_region }}"
+        aws_access_key_id: "{{ cloud_artifact_storage_accountname }}"
+        aws_secret_access_key: "{{ cloud_artifact_storage_secret }}"
+      when: cloud_service_provider == "aws"
+
+    - name: upload artifact to oci oss
+      include_role:
+        name: oci-cloud-storage
+        apply:
+          environment:
+           OCI_CLI_AUTH: "instance_principal"
+        tasks_from: upload.yml
+      vars:
+        local_file_or_folder_path: "{{ artifact_path }}"
+        oss_bucket_name: "{{ cloud_storage_artifacts_bucketname }}"
+        oss_path: "{{ artifact }}"
+      when: cloud_service_provider == "oci"

ansible/cassandra-backup.yml

@@ -2,8 +2,5 @@
   become: yes
   vars_files:
     - ['{{inventory_dir}}/secrets.yml']
-  environment:
-    AZURE_STORAGE_ACCOUNT: "{{sunbird_management_storage_account_name}}"
-    AZURE_STORAGE_KEY: "{{sunbird_management_storage_account_key}}"
   roles:
    - cassandra-backup

ansible/cassandra-restore.yml

@@ -3,8 +3,5 @@
   gather_facts: no
   vars_files:
     - ['{{inventory_dir}}/secrets.yml']
-  environment:
-    AZURE_STORAGE_ACCOUNT: "{{sunbird_management_storage_account_name}}"
-    AZURE_STORAGE_KEY: "{{sunbird_management_storage_account_key}}"
   roles:
    - cassandra-restore

ansible/es_backup.yml

@@ -4,19 +4,29 @@
   vars_files:
     - ['{{inventory_dir}}/secrets.yml']
   tasks:
-    - name: Create a container for storing backups
-      command: az storage container create  --name elasticsearch-snapshots --public-access blob
-      environment:
-        AZURE_STORAGE_ACCOUNT: "{{sunbird_management_storage_account_name}}"
-        AZURE_STORAGE_KEY: "{{sunbird_management_storage_account_key}}"
-
+    - name: Ensure azure blob storage container exists
+      include_role:
+        name: azure-cloud-storage
+        tasks_from: container-create.yml
+      vars:
+        blob_container_name: "elasticsearch-snapshots"
+        container_public_access: "off"
+        storage_account_name: "{{ cloud_management_storage_accountname }}"
+        storage_account_key: "{{ cloud_management_storage_secret }}"
+      when: cloud_service_provider == "azure"
 
 - hosts: composite-search-cluster
   become: yes
   vars_files:
     - ['{{inventory_dir}}/secrets.yml']
   roles:
-    - es-azure-snapshot
+    - role: es-azure-snapshot
+      when: cloud_service_provider == "azure"
+    - role: es-s3-snapshot
+      when: cloud_service_provider == "aws"
+    - role: es-gcs-snapshot
+      when: cloud_service_provider == "gcloud"
+    - role: es5-snapshot-purge
   tags:
     - es_backup
   run_once: true

ansible/inventory/env/group_vars/all.yml

@@ -85,8 +85,8 @@ search_es7_host: "{{ groups['es7']|join(':9200,')}}:9200"
 
 mlworkbench: "{{ groups['mlworkbench'][0]}}"
 
-azure_account: "{{ sunbird_public_storage_account_name }}"
-azure_secret: "{{ sunbird_public_storage_account_key }}"
+azure_account: "{{ cloud_public_storage_accountname }}"
+azure_secret: "{{ cloud_public_storage_secret }}"
 dedup_redis_host: "{{ dp_redis_host }}"
 kp_redis_host: "{{ groups['redisall'][0] }}"
 neo4j_route_path: "bolt://{{ groups['learning-neo4j-node1'][0] }}:7687"
@@ -97,6 +97,7 @@ middleware_hierarchy_keyspace: "{{ instance }}_hierarchy_store"
 kp_print_service_base_url: "http://{{private_ingressgateway_ip}}/print"
 cert_reg_service_base_url: "http://{{private_ingressgateway_ip}}/certreg"
 kp_search_service_base_url: "http://{{private_ingressgateway_ip}}/search"
+kp_dial_service_base_url: "http://{{private_ingressgateway_ip}}/dial"
 lms_service_base_url: "http://{{private_ingressgateway_ip}}/lms"
 learner_service_base_url: "http://{{private_ingressgateway_ip}}/learner"
 sourcing_content_service_base_url: "http://{{dock_private_ingressgateway_ip | default('localhost') }}/content"
@@ -116,3 +117,13 @@ cert_azure_storage_key: "{{sunbird_private_storage_account_name}}"
 default_channel: "org.sunbird"
 download_neo4j: true
 neo4j_upstream_download: false
+enable_suppress_exception: false
+enable_rc_certificate: true
+
+# SB-31155
+plugin_storage: "{{ plugin_container_name }}"
+
+cloud_storage_endpoint: "{{ cloud_public_storage_endpoint }}"
+cloudstorage_relative_path_prefix_content: "CONTENT_STORAGE_BASE_PATH"
+cloudstorage_relative_path_prefix_dial: "DIAL_STORAGE_BASE_PATH"
+cloudstorage_metadata_list: '["appIcon", "artifactUrl", "posterImage", "previewUrl", "thumbnail", "assetsMap", "certTemplate", "itemSetPreviewUrl", "grayScaleAppIcon", "sourceURL", "variants", "downloadUrl", "streamingUrl", "toc_url", "data", "question", "solutions", "editorState", "media", "pdfUrl", "transcripts"]'

ansible/lp_neo4j-backup.yml

@@ -1,9 +1,6 @@
 - hosts: learning-neo4j-node1  #if it is a cluster learning-neo4j-node1 should be always master node
   vars_files:
     - "{{inventory_dir}}/secrets.yml"
-  environment:
-    AZURE_STORAGE_ACCOUNT: "{{sunbird_management_storage_account_name}}"
-    AZURE_STORAGE_KEY: "{{sunbird_management_storage_account_key}}"
   become: yes
   become_user: "{{ learner_user }}"
   roles: