Skip to content

Commit c653141

Browse files
vinukumar-vsgitbook-bot
vinukumar-vs
authored and
gitbook-bot
committed
GitBook: [#466] Added Note section
1 parent ffbd962 commit c653141

File tree

2 files changed

+33
-18
lines changed

2 files changed

+33
-18
lines changed

‎learn/product-and-developer-guide/other/data-migration.md

+30-15
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,11 @@ migration flow diagram (migration-job flow & migration-process workflow)
6666
{% hint style="info" %}
6767
Note: 
6868

69-
1. Jenkins Job '<mark style="color:green;">**Neo4jElasticSearchSyncTool**</mark>' is used to insert the events into 'csp-migrator' job input topic. 'csp-migrator' job will further insert topics into 'live-node-publisher' job and 'live-video-stream-generator' jobs based on conditions. Jenkins job command: **migratecspdata**
70-
2. '**cassandra-data-migration'** job is to be used for migration of '_dialcode\_images'_ and '_dialcode\_batch'_ cassandra tables in '_dialcodes_**'** keyspace.
71-
3. <mark style="color:orange;">**We suggest to run the migration flink jobs in a separate kafka setup with increased processing ability and storage for storing all kakfa events and logs.**</mark>
69+
* Jenkins Job '<mark style="color:green;">**Neo4jElasticSearchSyncTool**</mark>' is used to insert the events into 'csp-migrator' job input topic. 'csp-migrator' job will further insert topics into 'live-node-publisher' job and 'live-video-stream-generator' jobs based on conditions. Jenkins job command: **migratecspdata**
70+
* '**cassandra-data-migration'** job is to be used for migration of '_dialcode\_images'_ and '_dialcode\_batch'_ cassandra tables in '_dialcodes_**'** keyspace.
71+
* <mark style="color:orange;">**Run the migration flink jobs in a separate kafka setup with increased processing ability and storage for storing all kakfa events and logs.**</mark>
72+
* <mark style="color:orange;">**Increase the infra for neo4j. Also, increase the neo4j max heap size in neo4j conf file.**</mark>&#x20;
73+
* <mark style="color:orange;">**Increase infra for logstash, search-indexer flink job and ElasticSearch to handle the neo4j transaction data sync.**</mark>
7274
{% endhint %}
7375

7476
The content migration should execute in the below order only. Otherwise there is a chances of migration failure because of dependent content is not yet migrated. [more details](https://docs.google.com/spreadsheets/d/13DaXCx8uToOwinlAPxvTat8NELxiPgG4KXATcKaJm\_c/edit#gid=1675310401\&range=K3)
@@ -82,18 +84,31 @@ The content migration should execute in the below order only. Otherwise there is
8284

8385
| Sequence | Type | Sync Tool Jenkins Parameters |
8486
| -------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
85-
| 1 | Video Asset | --graphId domain --objectType Asset --mimeType video/webm,video/mp4 --limit 1000 |
86-
| 2 | Other Asset | --graphId domain --objectType Asset --limit 1000 |
87-
| 3 | Video Content | --graphId domain --objectType Content,ContentImage --mimeType video/mp4,video/webm -limit 1000 |
88-
| 4 | Plugin, Youtube Content, PDF Content,EPUB Content | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.plugin-archive,video/x-youtube,application/pdf,application/epub --limit 1000 |
89-
| 5 | AssessmentItem | --graphId domain --objectType AssessmentItem --limit 1000 |
90-
| 6 | ItemSet | --graphId domain --objectType ItemSet --limit 1000 |
91-
| 7 | Question | --graphId domain --objectType Question -limit 1000 |
92-
| 8 | QuestionSet | --graphId domain --objectType QuestionSet,QuestionSetImage -limit 1000 |
93-
| 9 | H5P Content | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.h5p-archive -limit 1000 |
94-
| 10 | HTML | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.html-archive -limit 1000 |
95-
| 11 | ECML | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.ecml-archive -limit 1000 |
96-
| 12 | Collection | --graphId domain --objectType Collection,CollectionImage --mimeType application/vnd.ekstep.content-collection -limit 1000 |
87+
| 1 | Video Asset | --graphId domain --objectType Asset --mimeType video/webm,video/mp4 --delay 2000 |
88+
| 2 | Other Asset | --graphId domain --objectType Asset --delay 2000 |
89+
| 3 | Video Content | --graphId domain --objectType Content,ContentImage --mimeType video/mp4,video/webm --delay 2000 |
90+
| 4 | Plugin, Youtube Content, PDF Content,EPUB Content | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.plugin-archive,video/x-youtube,application/pdf,application/epub --delay 2000 |
91+
| 5 | AssessmentItem | --graphId domain --objectType AssessmentItem --delay 2000 |
92+
| 6 | ItemSet | --graphId domain --objectType ItemSet --delay 2000 |
93+
| 7 | H5P Content | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.h5p-archive --delay 2000 |
94+
| 8 | HTML | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.html-archive --delay 2000 |
95+
| 9 | ECML | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.ecml-archive --delay 2000 |
96+
| 10 | Collection | --graphId domain --objectType Collection,CollectionImage --mimeType application/vnd.ekstep.content-collection --delay 2000 |
97+
98+
{% hint style="info" %}
99+
Note:
100+
101+
* ECML migration can be triggered only after Asset, AssessmentItem and ItemSet migration is completed.
102+
* For Collection migration to be triggered, pre-requisites are:
103+
104+
&#x20; a. Question, QuestionSet migration should be completed (as part of Inquiry BB).
105+
106+
&#x20; b. All assets and contents are to be migrated successfully.&#x20;
107+
108+
&#x20; c. All migrated contents data should be synced to ElasticSearch.
109+
{% endhint %}
110+
111+
97112

98113
### Migration status: migrationVersion of the node object
99114

0 commit comments

Comments
 (0)