Merge pull request #386 from marklogic/release/2.5.0

Merge release/2.5.0 into master

Author: rjrudin

.env

@@ -1,3 +1,3 @@
 # Defines environment variables for docker-compose.
 # Can be overridden via e.g. `MARKLOGIC_TAG=latest-10.0 docker-compose up -d --build`.
-MARKLOGIC_TAG=11.3.0-ubi
+MARKLOGIC_TAG=ml-docker-db-dev-tierpoint.bed-artifactory.bedford.progress.com/marklogic/marklogic-server-ubi:latest-12

CONTRIBUTING.md

@@ -21,9 +21,6 @@ If you wish to use `docker compose`, perform the following steps before deployin
 
 The above will result in a new MarkLogic instance with a single node. 
 
-Alternatively, if you would like to test against a 3-node MarkLogic cluster with a load balancer in front of it, 
-run `docker compose -f docker-compose-3nodes.yaml up -d --build`.
-
 ## Deploying the test application
 
 To deploy the test application, first create `./gradle-local.properties` and add the following to it:
@@ -59,18 +56,18 @@ To configure the SonarQube service, perform the following steps:
 7. Click on "Use the global setting" and then "Create project".
 8. On the "Analysis Method" page, click on "Locally".
 9. In the "Provide a token" panel, click on "Generate". Copy the token.
-10. Add `systemProp.sonar.token=your token pasted here` to `gradle-local.properties` in the root of your project, creating
+10. Add `systemProp.sonar.login=your token pasted here` to `gradle-local.properties` in the root of your project, creating
 that file if it does not exist yet.
 
 To run SonarQube, run the following Gradle tasks using Java 17, which will run all the tests with code coverage and 
 then generate a quality report with SonarQube:
 
     ./gradlew test sonar
 
-If you do not add `systemProp.sonar.token` to your `gradle-local.properties` file, you can specify the token via the
+If you do not add `systemProp.sonar.login` to your `gradle-local.properties` file, you can specify the token via the
 following:
 
-    ./gradlew test sonar -Dsonar.token=paste your token here
+    ./gradlew test sonar -Dsonar.login=paste your token here
 
 When that completes, you will see a line like this near the end of the logging:
 
@@ -87,25 +84,6 @@ You can also force Gradle to run `sonar` if any tests fail:
 
     ./gradlew clean test sonar --continue
 
-## Accessing MarkLogic logs in Grafana
-
-This project's `docker-compose-3nodes.yaml` file includes
-[Grafana, Loki, and promtail services](https://grafana.com/docs/loki/latest/clients/promtail/) for the primary reason of
-collecting MarkLogic log files and allowing them to be viewed and searched via Grafana.
-
-Once you have run `docker compose`, you can access Grafana at http://localhost:3000 . Follow these instructions to
-access MarkLogic logging data:
-
-1. Click on the hamburger in the upper left hand corner and select "Explore", or simply go to
-   http://localhost:3000/explore .
-2. Verify that "Loki" is the default data source - you should see it selected in the upper left hand corner below
-   the "Home" link.
-3. Click on the "Select label" dropdown and choose `job`. Click on the "Select value" label for this filter and
-   select `marklogic` as the value.
-4. Click on the blue "Run query" button in the upper right hand corner.
-
-You should now see logs from all 3 nodes in the MarkLogic cluster.
-
 # Testing with PySpark
 
 The documentation for this project 
@@ -120,7 +98,7 @@ This will produce a single jar file for the connector in the `./build/libs` dire
 
 You can then launch PySpark with the connector available via:
 
-    pyspark --jars build/libs/marklogic-spark-connector-2.4-SNAPSHOT.jar
+    pyspark --jars marklogic-spark-connector/build/libs/marklogic-spark-connector-2.5-SNAPSHOT.jar
 
 The below command is an example of loading data from the test application deployed via the instructions at the top of 
 this page. 
@@ -164,7 +142,7 @@ For a quick test of writing documents, use the following:
 
 ```
 
-spark.read.option("header", True).csv("src/test/resources/data.csv")\
+spark.read.option("header", True).csv("marklogic-spark-connector/src/test/resources/data.csv")\
     .repartition(2)\
     .write.format("marklogic")\
     .option("spark.marklogic.client.uri", "spark-test-user:spark@localhost:8000")\
@@ -196,7 +174,7 @@ The Spark master GUI is at <http://localhost:8080>. You can use this to view det
 
 Now that you have a Spark cluster running, you just need to tell PySpark to connect to it:
 
-    pyspark --master spark://NYWHYC3G0W:7077 --jars build/libs/marklogic-spark-connector-2.4-SNAPSHOT.jar
+    pyspark --master spark://NYWHYC3G0W:7077 --jars marklogic-spark-connector/build/libs/marklogic-spark-connector-2.5-SNAPSHOT.jar
 
 You can then run the same commands as shown in the PySpark section above. The Spark master GUI will allow you to 
 examine details of each of the commands that you run.
@@ -215,12 +193,16 @@ You will need the connector jar available, so run `./gradlew clean shadowJar` if
 You can then run a test Python program in this repository via the following (again, change the master address as 
 needed); note that you run this outside of PySpark, and `spark-submit` is available after having installed PySpark:
 
-    spark-submit --master spark://NYWHYC3G0W:7077 --jars build/libs/marklogic-spark-connector-2.4-SNAPSHOT.jar src/test/python/test_program.py
+    spark-submit --master spark://NYWHYC3G0W:7077 --jars marklogic-spark-connector/build/libs/marklogic-spark-connector-2.5-SNAPSHOT.jar marklogic-spark-connector/src/test/python/test_program.py
 
 You can also test a Java program. To do so, first move the `com.marklogic.spark.TestProgram` class from `src/test/java`
-to `src/main/java`. Then run `./gradlew clean shadowJar` to rebuild the connector jar. Then run the following:
+to `src/main/java`. Then run the following:
 
-    spark-submit --master spark://NYWHYC3G0W:7077 --class com.marklogic.spark.TestProgram build/libs/marklogic-spark-connector-2.4-SNAPSHOT.jar
+```
+./gradlew clean shadowJar
+cd marklogic-spark-connector
+spark-submit --master spark://NYWHYC3G0W:7077 --class com.marklogic.spark.TestProgram build/libs/marklogic-spark-connector-2.5-SNAPSHOT.jar
+```
 
 Be sure to move `TestProgram` back to `src/test/java` when you are done. 
 

Dockerfile

@@ -6,13 +6,12 @@ def runtests(String javaVersion){
     export GRADLE_USER_HOME=$WORKSPACE/$GRADLE_DIR
     export PATH=$GRADLE_USER_HOME:$JAVA_HOME/bin:$PATH
     cd marklogic-spark-connector
-    echo "mlPassword=admin" > gradle-local.properties
     echo "Waiting for MarkLogic server to initialize."
     sleep 30s
    ./gradlew -i mlDeploy
    echo "Loading data a second time to try to avoid Optic bug with duplicate rows being returned."
    ./gradlew -i mlLoadData
-   ./gradlew test || true
+   ./gradlew clean testCodeCoverageReport || true
   '''
   junit '**/build/**/*.xml'
 }
@@ -55,10 +54,10 @@ pipeline{
         sh label:'mlsetup', script: '''#!/bin/bash
             echo "Removing any running MarkLogic server and clean up MarkLogic data directory"
             sudo /usr/local/sbin/mladmin remove
+            docker-compose down -v || true
             sudo /usr/local/sbin/mladmin cleandata
             cd marklogic-spark-connector
             mkdir -p docker/marklogic/logs
-            docker-compose down -v || true
             docker-compose up -d --build
           '''
         runtests('JAVA17_HOME_DIR')
@@ -108,7 +107,7 @@ pipeline{
                 cd marklogic-spark-connector
                 mkdir -p docker/marklogic/logs
                 docker-compose down -v || true
-                MARKLOGIC_TAG=latest-10.0 docker-compose up -d --build
+                MARKLOGIC_TAG=progressofficial/marklogic-db:latest-11 docker-compose up -d --build
             '''
             runtests('JAVA17_HOME_DIR')
       }
@@ -117,6 +116,7 @@ pipeline{
           sh label:'mlcleanup', script: '''#!/bin/bash
             cd marklogic-spark-connector
             docker-compose down -v || true
+            sudo /usr/local/sbin/mladmin delete $WORKSPACE/marklogic-spark-connector/docker/caddy/
             sudo /usr/local/sbin/mladmin delete $WORKSPACE/marklogic-spark-connector/docker/marklogic/logs/
           '''
         }

Jenkinsfile

@@ -13,6 +13,7 @@ Third Party Notices
 jackson-dataformat-xml 2.15.2 (Apache-2.0)
 jdom2 2.0.6.1 (Apache-2.0)
 jena-arq 4.10.0 (Apache-2.0)
+langchain4j 0.35.0 (Apache-2.0)
 marklogic-client-api 7.0.0 (Apache-2.0)
 okhttp 4.12.0 (Apache-2.0)
 
@@ -22,7 +23,7 @@ Apache License 2.0 (Apache-2.0)
 
 Third-Party Components
 
-The following is a list of the third-party components used by the MarkLogic® Spark connector 2.4.2 (last updated October 17, 2024):
+The following is a list of the third-party components used by the MarkLogic® Spark connector 2.5.0 (last updated December 17, 2024):
 
 jackson-dataformat-xml 2.15.2 (Apache-2.0)
 https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-xml/
@@ -36,6 +37,10 @@ jena-arq 4.10.0 (Apache-2.0)
 https://repo1.maven.org/maven2/org/apache/jena/jena-arq/
 For the full text of the Apache-2.0 license, see Apache License 2.0 (Apache-2.0)
 
+langchain4j 0.35.0 (Apache-2.0)
+https://repo1.maven.org/maven2/dev/langchain4j/langchain4j/
+For the full text of the Apache-2.0 license, see Apache License 2.0 (Apache-2.0)
+
 marklogic-client-api 7.0.0 (Apache-2.0)
 https://repo1.maven.org/maven2/com/marklogic/marklogic-client-api/
 For the full text of the Apache-2.0 license, see Apache License 2.0 (Apache-2.0)