Skip to content

DEVELOPER_GUIDE.md

Latest commit

 

History

History
155 lines (117 loc) · 5.47 KB

DEVELOPER_GUIDE.md

File metadata and controls

155 lines (117 loc) · 5.47 KB

Development Guide

Table of Contents

Prerequisites

  • Java Development Kit (JDK) 17+
  • Python3
  • Docker/Minikube/K3s/etc (for local deployment)
  • Helm
  • AWS CLI (for AWS deployment)
  • Node.js v22 (downloaded automatically by Gradle)
  • AWS Cloud Development Kit (CDK) (for AWS deployment, downloaded automatically by Gradle)

Kubernetes Quick Start

  • This Kubernetes Guide shows you how to create a minikube cluster locally and deploy the Migration Assistant to it.
  • This AWS EKS Guide shows you how to deploy an EKS cluster and deploy the Migration Assistant to it.

See the project wiki to learn more about how to use the migration console and its workflow commands.

Project Structure

  • CreateSnapshot: Tools for creating cluster snapshots.
  • DocumentsFromSnapshotMigration: Utilities for migrating documents from snapshots.
  • MetadataMigration: Core functionality for migrating cluster metadata.
  • RFS (Reindex-From-Snapshot):
    • Migration utilities for document reindexing and metadata migration.
    • Includes tracing contexts for both document and metadata migrations.
  • TrafficCapture (Capture-and-Replay): Projects for proxying, capturing, and replaying HTTP traffic.
  • migrationConsole: A comprehensive CLI tool for executing the migration workflow.
  • deployment: AWS deployment scripts and configurations.
  • dev-tools: Development utilities and API request templates.
  • docs: Project documentation and architecture diagrams.
  • libraries: Shared libraries used across the project.
  • test: End-to-end testing scripts and configurations.
  • transformation: Data transformation utilities for migration processes.
  • dashboardsSanitizer: CLI tool for sanitizing dashboard configurations.
  • testHelperFixtures: Test utilities including HTTP client for testing.

The migration console CLI provides users with a centralized interface to execute and manage the entire migration workflow, including:

  • Configuring source and target clusters
  • Managing backfill operations
  • Controlling traffic replay
  • Monitoring migration progress through metrics
  • Handling snapshots and metadata
  • Integrating with various deployment environments (Docker locally, AWS ECS, EKS, K8s)

Users can interact with the migration process through the CLI, which orchestrates the different components of the migration toolkit to perform a seamless migration between Elasticsearch and OpenSearch clusters.

Building the Project

./gradlew build

Running Tests

./gradlew test

Building Images

Build images with buildkit and jib. See buildImages for instructions to set that up.

./gradlew :buildImages:buildImagesToRegistry

Running the Project

Code Style

We use Spotless for code formatting. To check and apply the code style:

./gradlew spotlessCheck
./gradlew spotlessApply

Pre-Commit Hooks

Install the pre-commit hooks:

./install_githooks.sh

Publishing Images

This project can be published to a local Maven repository with:

./gradlew publishToMavenLocal

And subsequently imported into a separate Gradle project with (replacing name with any subProject name):

repositories {
    mavenCentral()
    mavenLocal()
}

dependencies {
    implementation group: "org.opensearch.migrations.trafficcapture", name: "captureKafkaOffloader", version: "0.1.0-SNAPSHOT"
    //... other dependencies
}

The entire list of published subprojects can be viewed as follows:

./gradlew listPublishedArtifacts

To include a test Fixture dependency, define the import similar to the following:

testImplementation testFixtures('org.opensearch.migrations.trafficcapture:trafficReplayer:0.1.0-SNAPSHOT')

Development Environments

VSCode

Python

Settings.json files are already set up in the project, make sure that venv environments are created in all folders with pipfile. Bootstrap your environment by running the following command that creates all the environments.

find . -name Pipfile -not -path "*/cdk.out/*"  | while read pipfile; do
  dir=$(dirname "$pipfile")
  echo "Setting up .venv in $dir"
  (cd "$dir" && PIPENV_IGNORE_VIRTUALENVS=1 PIPENV_VENV_IN_PROJECT=1 pipenv install)
done