add benchmarking scripts for parquet loader

CHANGELOG.md

@@ -4,30 +4,10 @@ All notable changes to this project are documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). See the [CONTRIBUTING guide](./CONTRIBUTING.md#Changelog) for instructions on how to add changelog entries.
 
-## [Unreleased 3.2](https://github.com/opensearch-project/opensearch-jvector/compare/2.x...HEAD)
-### Features
-### Enhancements
-* PQ refinement during merge [109](https://github.com/opensearch-project/opensearch-jvector/issues/109)
-* Persistent Ordinal To docID Mapping [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
-* Incremental Insertion With Leading Segment [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
-* Remove Redundant FlatVectorFormat [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
-* Remove Redundant DocValuesFormat [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
-### Bug Fixes
-* Fix for sorted indices [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
-* Fix for missing fields [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
-### Infrastructure
-* Upgrade to JDK24 [165] (https://github.com/opensearch-project/opensearch-jvector/pull/165)
-* Upgrade Gradle to 8.14 [165] (https://github.com/opensearch-project/opensearch-jvector/pull/165)
-### Documentation
-* Add docker instructions [163] (https://github.com/opensearch-project/opensearch-jvector/pull/163)
-### Maintenance
-* Fix documentation bugs [161] (https://github.com/opensearch-project/opensearch-jvector/pull/161)
-### Refactoring
-* Remove jVector Codec [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
-
-## [Unreleased 2.x](https://github.com/opensearch-project/opensearch-jvector/compare/2.18...2.x)
+## [Unreleased 3.3](https://github.com/opensearch-project/opensearch-jvector/compare/3.2...HEAD)
 ### Features
 ### Enhancements
+* Add script for loading vector data using Parquet [192](https://github.com/opensearch-project/opensearch-jvector/issues/192)
 ### Bug Fixes
 ### Infrastructure
 ### Documentation

scripts/parquet-loader/README.md

@@ -0,0 +1,49 @@
+# OpenSearch Parquet Loader
+
+This script efficiently loads data from Parquet files into an OpenSearch index, leveraging batch processing and multiple connections to maximize indexing speed.
+
+## Prerequisites
+
+- Python 3.8+
+- Access to an OpenSearch cluster
+
+## Setup
+
+1. **Create a python virtual environment:**
+   ```bash
+   sudo apt install python3.11-venv
+   # Using venv (Python 3.3+)
+   python3 -m venv .venv
+
+   # Activate the virtual environment
+   # On Windows:
+   .venv\Scripts\activate
+   # On macOS/Linux:
+   source .venv/bin/activate
+   ```
+
+2.  **Install dependencies:**
+    ```bash
+    pip install -r requirements.txt
+    ```
+
+3.  **Configure environment variables:**
+    Create a `.env` file in the project root and add your OpenSearch connection details:
+    ```
+    OPENSEARCH_HOSTS='["http://localhost:9200"]'
+    OPENSEARCH_INDEX="my-index"
+    ```
+    Alternatively, set these variables on your local terminal
+    export OPENSEARCH_HOSTS='[{"host":"localhost","port":9200,"scheme":"http"}]'
+    export OPENSEARCH_INDEX="my-index"
+
+## Usage
+
+Run the script from the command line, providing the path to your Parquet file and the number of parallel workers:
+
+```bash
+python osbench.py --path /path/to/your/data.parquet --procs 4
+```
+
+-   `/path/to/your/data.parquet`: The Parquet file to load.
+-   `--workers`: The number of parallel connections to use.

scripts/parquet-loader/create_index.sh

@@ -0,0 +1,39 @@
+curl -X PUT "https://localhost:9200/jvector-index?pretty" --insecure -H 'Content-Type: application/json' -d'
+{ 
+  "settings": { 
+    "index": { 
+      "knn": true, 
+      "refresh_interval":  -1, 
+      "number_of_replicas": 0, 
+      "number_of_shards": 1, 
+      "merge": { 
+         "policy": { 
+            "max_merged_segment": "50g" 
+          } 
+       } 
+    } 
+  }, 
+  "mappings": { 
+    "_source": { 
+       "excludes": ["embeddings", "chunk_id"], 
+        "recovery_source_excludes": ["embeddings", "chunk_id"] 
+     }, 
+    "properties": { 
+      "chunk_id": {"type": "long"}, 
+      "embeddings": { 
+        "type": "knn_vector", 
+        "method": { 
+          "name": "disk_ann", 
+          "space_type": "l2", 
+          "engine": "jvector", 
+          "parameters": { 
+            "m": 32, 
+            "ef_construction": 200, 
+            "advanced.num_pq_subspaces": 48 
+          } 
+        }, 
+        "dimension": 384 
+      } 
+    } 
+  } 
+}' -u admin