feat: Jira v1 to v2 migration (#399)

* Jira V2 migration

* Fix error when fetching project with no issues

* Add documentation and debug logs

* Jira v2 integration (e2e) tests

* Jira integration test

* Unit tests

* Lint fix

* Version and changelog update; jira v1 to v2 migration

* Lint fix

* Enable jira integration test in ci

* Fix formatting

* align with v2 config

* TEMP: disable google-drive e2e test

* Docs update

* Revert "TEMP: disable google-drive e2e test"

This reverts commit e14845d.

* Rever changelog formatting

* After review fixes

* Review fixes

* lint fix

* Remove old e2e jira test

Author: mpolomdeepsense

.github/workflows/e2e.yml

@@ -278,6 +278,9 @@ jobs:
         CONFLUENCE_API_TOKEN: ${{secrets.CONFLUENCE_API_TOKEN}}
         # Notion
         NOTION_API_KEY: ${{ secrets.NOTION_API_KEY }}
+        # Jira
+        JIRA_INGEST_API_TOKEN: ${{ secrets.JIRA_INGEST_API_TOKEN }}
+        JIRA_INGEST_USER_EMAIL: ${{ secrets.JIRA_INGEST_USER_EMAIL }}
       run : |
         source .venv/bin/activate
         make install-test

CHANGELOG.md

@@ -1,3 +1,10 @@
+## 0.5.10-dev0
+
+### Enhancements
+
+* **Migrate Jira Source connector from V1 to V2**
+* **Add Jira Source connector integration and unit tests**
+
 ## 0.5.9
 
 ### Features

test/integration/connectors/expected_results/jira/directory_structure.json

@@ -0,0 +1,12 @@
+{
+  "directory_structure": [
+    "JCTP1/10000.txt",
+    "JCTP1/10001.txt",
+    "JCTP1/10002.txt",
+    "JCTP1/10013.txt",
+    "JCTP2/10003.txt",
+    "JCTP2/10004.txt",
+    "JCTP2/10005.txt",
+    "JCTP3/10014.txt"
+  ]
+}

test/integration/connectors/expected_results/jira/downloads/JCTP1/10000.txt

@@ -0,0 +1,58 @@
+IssueID_IssueKey:10000     JCTP1-1
+
+
+
+ProjectID_Key:JCTP1     Jira Connector Test Project 1
+
+
+
+
+        IssueType:Task
+        
+
+        Status:To Do
+        
+
+        Priority:{'self': 'https://unstructured-jira-connector-test.atlassian.net/rest/api/2/priority/3', 'iconUrl': 'https://unstructured-jira-connector-test.atlassian.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}
+        
+
+        AssigneeID_Name:712020:7bc7fdcb-67e7-435d-b4a2-128aee12820c     Unstructured Devops
+        
+
+        ReporterAdr_Name:[email protected]     Unstructured Devops
+        
+
+        Labels:Label1     Label2
+        
+
+        Components:
+        
+
+        
+
+
+
+
+
+Unstructured Devops     My comment 1
+     Unstructured Devops     My attachment image lorem ipsum:
+
+!image-20230823-143650.png|width=83.33333333333333%!
+
+
+
+
+        Test todo 1
+        
+
+        Lorem ipsum dolor sit amet, consectetur adipiscing elit. [Nam quid possumus facere melius?|http://loripsum.net/] Ita relinquet duas, de quibus etiam atque etiam consideret. Quo modo autem philosophus loquitur? Quid est enim aliud esse versutum? His enim rebus detractis negat se reperire in asotorum vita quod reprehendat. Non est ista, inquam, Piso, magna dissensio. Duo Reges: constructio interrete. In eo enim positum est id, quod dicimus esse expetendum. Traditur, inquit, ab Epicuro ratio neglegendi doloris. Negat enim summo bono afferre incrementum diem. Aberat omnis dolor, qui si adesset, nec molliter ferret et tamen medicis plus quam philosophis uteretur.
+
+Sedulo, inquam, faciam. Ergo, si semel tristior effectus est, hilara vita amissa est? Quamquam tu hanc copiosiorem etiam soles dicere. An eum locum libenter invisit, ubi Demosthenes et Aeschines inter se decertare soliti sunt? _Quippe: habes enim a rhetoribus;_ Non minor, inquit, voluptas percipitur ex vilissimis rebus quam ex pretiosissimis. Ut in geometria, prima si dederis, danda sunt omnia. Negat enim summo bono afferre incrementum diem.
+
+[Et nemo nimium beatus est;|http://loripsum.net/] Nam et complectitur verbis, quod vult, et dicit plane, quod intellegam; Ab his oratores, ab his imperatores ac rerum publicarum principes extiterunt. Ergo adhuc, quantum equidem intellego, causa non videtur fuisse mutandi nominis. Quis enim redargueret? Ita fit cum gravior, tum etiam splendidior oratio. Sed ut iis bonis erigimur, quae expectamus, sic laetamur iis, quae recordamur. _Bork_ Tubulum fuisse, qua illum, cuius is condemnatus est rogatione, P. [Eiuro, inquit adridens, iniquum, hac quidem de re;|http://loripsum.net/] Si quae forte-possumus.
+        
+
+        https://unstructured-jira-connector-test.atlassian.net/rest/api/2/attachment/10000
+        
+
+        

test/integration/connectors/expected_results/jira/downloads/JCTP1/10001.txt

@@ -0,0 +1,55 @@
+IssueID_IssueKey:10001     JCTP1-2
+
+
+
+ProjectID_Key:JCTP1     Jira Connector Test Project 1
+
+
+
+
+        IssueType:Task
+        
+
+        Status:In Progress
+        
+
+        Priority:{'self': 'https://unstructured-jira-connector-test.atlassian.net/rest/api/2/priority/3', 'iconUrl': 'https://unstructured-jira-connector-test.atlassian.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}
+        
+
+        AssigneeID_Name:{}     {}
+        
+
+        ReporterAdr_Name:[email protected]     Unstructured Devops
+        
+
+        Labels:
+        
+
+        Components:
+        
+
+        
+
+
+
+
+
+Unstructured Devops     My comment 1
+
+
+
+
+        Test in progress 1
+        
+
+        Cur deinde Metrodori liberos commendas? Quicquid porro animo cernimus, id omne oritur a sensibus; Ergo ita: non posse honeste vivi, nisi honeste vivatur? [Quis Aristidem non mortuum diligit?|http://loripsum.net/] Quid, de quo nulla dissensio est? [Bork|http://loripsum.net/]
+
+Atqui pugnantibus et contrariis studiis consiliisque semper utens nihil quieti videre, nihil tranquilli potest. Habes, inquam, Cato, formam eorum, de quibus loquor, philosophorum. [Ecce aliud simile dissimile.|http://loripsum.net/] Ad quorum et cognitionem et usum iam corroborati natura ipsa praeeunte deducimur. Itaque et manendi in vita et migrandi ratio omnis iis rebus, quas supra dixi, metienda. Quicquid enim a sapientia proficiscitur, id continuo debet expletum esse omnibus suis partibus; At ille non pertimuit saneque fidenter: Istis quidem ipsis verbis, inquit; Utrum igitur tibi litteram videor an totas paginas commovere?
+
+Duo Reges: constructio interrete. Nunc omni virtuti vitium contrario nomine opponitur. Non autem hoc: igitur ne illud quidem. Ergo hoc quidem apparet, nos ad agendum esse natos. Quo tandem modo? Quod si ita se habeat, non possit beatam praestare vitam sapientia. Propter nos enim illam, non propter eam nosmet ipsos diligimus.
+        
+
+        
+        
+
+        

test/integration/connectors/expected_results/jira/downloads/JCTP1/10002.txt

@@ -0,0 +1,55 @@
+IssueID_IssueKey:10002     JCTP1-3
+
+
+
+ProjectID_Key:JCTP1     Jira Connector Test Project 1
+
+
+
+
+        IssueType:Task
+        
+
+        Status:Done
+        
+
+        Priority:{'self': 'https://unstructured-jira-connector-test.atlassian.net/rest/api/2/priority/3', 'iconUrl': 'https://unstructured-jira-connector-test.atlassian.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}
+        
+
+        AssigneeID_Name:{}     {}
+        
+
+        ReporterAdr_Name:[email protected]     Unstructured Devops
+        
+
+        Labels:
+        
+
+        Components:
+        
+
+        
+
+
+
+
+
+Unstructured Devops     My comment 1
+
+
+
+
+        Test done 1
+        
+
+        _At iam decimum annum in spelunca iacet._ Quid ei reliquisti, nisi te, quoquo modo loqueretur, intellegere, quid diceret? Non laboro, inquit, de nomine. Duo Reges: constructio interrete. Nummus in Croesi divitiis obscuratur, pars est tamen divitiarum. Bork Itaque quantum adiit periculum! ad honestatem enim illum omnem conatum suum referebat, non ad voluptatem.
+
+[Bork|http://loripsum.net/] Illi enim inter se dissentiunt. Et non ex maxima parte de tota iudicabis? [Refert tamen, quo modo.|http://loripsum.net/] [Quam ob rem tandem, inquit, non satisfacit?|http://loripsum.net/] Ex quo, id quod omnes expetunt, beate vivendi ratio inveniri et comparari potest.
+
+Hic nihil fuit, quod quaereremus. Itaque haec cum illis est dissensio, cum Peripateticis nulla sane. Vos autem cum perspicuis dubia debeatis illustrare, dubiis perspicua conamini tollere. [Quae cum essent dicta, discessimus.|http://loripsum.net/] Nam, ut sint illa vendibiliora, haec uberiora certe sunt. [Equidem, sed audistine modo de Carneade?|http://loripsum.net/]
+        
+
+        
+        
+
+        

test/integration/connectors/expected_results/jira/downloads/JCTP1/10013.txt

@@ -0,0 +1,51 @@
+IssueID_IssueKey:10013     JCTP1-4
+
+
+
+ProjectID_Key:JCTP1     Jira Connector Test Project 1
+
+
+
+
+        IssueType:Subtask
+        
+
+        Status:To Do
+        
+
+        Priority:{'self': 'https://unstructured-jira-connector-test.atlassian.net/rest/api/2/priority/3', 'iconUrl': 'https://unstructured-jira-connector-test.atlassian.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}
+        
+
+        AssigneeID_Name:{}     {}
+        
+
+        ReporterAdr_Name:[email protected]     Unstructured Devops
+        
+
+        Labels:
+        
+
+        Components:
+        
+
+        
+
+
+
+
+
+Unstructured Devops     This is a Child Issue
+
+
+
+
+        ChildIssue1
+        
+
+        {}
+        
+
+        
+        
+
+        

test/integration/connectors/expected_results/jira/downloads/JCTP2/10003.txt

@@ -0,0 +1,50 @@
+IssueID_IssueKey:10003     JCTP2-1
+
+
+
+ProjectID_Key:JCTP2     Jira Connector Test Project 2
+
+
+
+
+        IssueType:Task
+        
+
+        Status:To Do
+        
+
+        Priority:{'self': 'https://unstructured-jira-connector-test.atlassian.net/rest/api/2/priority/3', 'iconUrl': 'https://unstructured-jira-connector-test.atlassian.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}
+        
+
+        AssigneeID_Name:712020:7bc7fdcb-67e7-435d-b4a2-128aee12820c     Unstructured Devops
+        
+
+        ReporterAdr_Name:[email protected]     Unstructured Devops
+        
+
+        Labels:
+        
+
+        Components:
+        
+
+        
+
+
+
+
+
+
+
+
+
+        Test Task 1
+        
+
+        {}
+        
+
+        
+        
+
+        

test/integration/connectors/expected_results/jira/downloads/JCTP2/10004.txt

@@ -0,0 +1,50 @@
+IssueID_IssueKey:10004     JCTP2-2
+
+
+
+ProjectID_Key:JCTP2     Jira Connector Test Project 2
+
+
+
+
+        IssueType:Task
+        
+
+        Status:In Progress
+        
+
+        Priority:{'self': 'https://unstructured-jira-connector-test.atlassian.net/rest/api/2/priority/3', 'iconUrl': 'https://unstructured-jira-connector-test.atlassian.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}
+        
+
+        AssigneeID_Name:712020:7bc7fdcb-67e7-435d-b4a2-128aee12820c     Unstructured Devops
+        
+
+        ReporterAdr_Name:[email protected]     Unstructured Devops
+        
+
+        Labels:
+        
+
+        Components:
+        
+
+        
+
+
+
+
+
+
+
+
+
+        Test Task 2
+        
+
+        {}
+        
+
+        
+        
+
+