From 6da2ad7fa0da8876bbcb86ec195d6c558bf28e69 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Fri, 9 May 2025 14:17:29 +0300 Subject: [PATCH 1/7] RDSC-3533 Update RDI documentation, part 1 --- content/embeds/rdi-db-reqs.md | 1 - content/embeds/rdi-vm-reqs.md | 2 +- .../installation/_index.md | 2 +- .../installation/install-k8s.md | 489 +++++------------- .../installation/install-vm.md | 222 +++----- .../installation/upgrade.md | 117 ++--- .../quick-start-guide.md | 10 +- 7 files changed, 254 insertions(+), 589 deletions(-) diff --git a/content/embeds/rdi-db-reqs.md b/content/embeds/rdi-db-reqs.md index 53a9533fd..e7575e7c0 100644 --- a/content/embeds/rdi-db-reqs.md +++ b/content/embeds/rdi-db-reqs.md @@ -3,7 +3,6 @@ quickstart or for development, 125MB and a single shard is sufficient. * If you are deploying RDI for a production environment then secure this database with a password and TLS. -* Provide the installation with the required RDI database details. * Set the database's [eviction policy]({{< relref "/operate/rs/databases/memory-performance/eviction-policy" >}}) to `noeviction`. Note that you can't set this using [`rladmin`]({{< relref "/operate/rs/references/cli-utilities/rladmin" >}}), diff --git a/content/embeds/rdi-vm-reqs.md b/content/embeds/rdi-vm-reqs.md index 51793f538..a8c3dad0e 100644 --- a/content/embeds/rdi-vm-reqs.md +++ b/content/embeds/rdi-vm-reqs.md @@ -1,7 +1,7 @@ * **CPU**: A minimum of 4 CPU cores. You should consider adding 2-6 extra cores on top of this if your dataset is big and you want to ingest the baseline snapshot as fast as possible. -* **RAM**: 2GB +* **RAM**: 8GB * **Disk**: On top of the OS footprint, RDI requires 20GB in the `/var` folder and 1GB in the `/opt` folder (to store the log files). This allows space for upgrades. diff --git a/content/integrate/redis-data-integration/installation/_index.md b/content/integrate/redis-data-integration/installation/_index.md index 23c04da10..8e34d2a28 100644 --- a/content/integrate/redis-data-integration/installation/_index.md +++ b/content/integrate/redis-data-integration/installation/_index.md @@ -18,6 +18,6 @@ weight: 2 --- The guides in this section explain the options you have for installing and upgrading RDI. -Before you use RDI, you must also configure your source database to use the CDC connector. See the +Before you use RDI, you must also configure your source database to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) section to learn how to do this. \ No newline at end of file diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index 100f36f0b..76a3a387f 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -23,42 +23,43 @@ to install on [Kubernetes](https://kubernetes.io/) (K8s). You can also The installation creates the following K8s objects: - A K8s [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) named `rdi`. -- [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the + You can also use a different namespace name if you prefer. +- [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) and + [services](https://kubernetes.io/docs/concepts/services-networking/service/) for the [RDI operator]({{< relref "/integrate/redis-data-integration/architecture#how-rdi-is-deployed" >}}), [metrics exporter]({{< relref "/integrate/redis-data-integration/observability" >}}), and API server. -- A [service account](https://kubernetes.io/docs/concepts/security/service-accounts/) along with a - [role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#restrictions-on-role-creation-or-update) - and [role binding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) for the RDI operator. -- A [Configmap](https://kubernetes.io/docs/concepts/configuration/configmap/) - for the different components with RDI Redis database details. +- A [service account](https://kubernetes.io/docs/concepts/security/service-accounts/) + and [RBAC resources](https://kubernetes.io/docs/reference/access-authn-authz/rbac) for the RDI operator. +- A [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) with RDI database details. - [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) - with the RDI Redis database credentials and TLS certificates. + with the RDI database credentials and TLS certificates. +- Other optional K8s resources such as [ingresses](https://kubernetes.io/docs/concepts/services-networking/ingress/) + that can be enabled depending on your K8s environment and needs. You can use this installation on [OpenShift](https://docs.openshift.com/) and other K8s distributions including cloud providers' K8s managed clusters. -You can pull the RDI images from the -[download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-1.6.7.tgz) +You can configure the RDI Helm chart to pull the RDI images from [dockerhub](https://hub.docker.com/u/redis) or from your own [private image registry](#using-a-private-image-registry). ## Before you install -Complete the following steps before running Helm: +Complete the following steps before installing the RDI Helm chart: - [Create the RDI database](#create-the-rdi-database) on your Redis Enterprise cluster. - Create a [user]({{< relref "/operate/rs/security/access-control/create-users" >}}) for the RDI database if you prefer not to use the default password (see [Access control]({{< relref "/operate/rs/security/access-control" >}}) for more information). -- Download the RDI helm chart tar file from the - [download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-1.6.7.tgz). +- Download the RDI Helm chart tar file from the + [Redis download center](https://redis.io/downloads/) (in the *Modules, Tools & Integration* category) . - If you want to use a private image registry, [prepare it with the RDI images](#using-a-private-image-registry). ### Create the RDI database RDI uses a database on your Redis Enterprise cluster to store its state -information. Use the Redis console to create the RDI database with the following +information. Use the Redis Enterprise Cluster Manager UI to create the RDI database with the following requirements: {{< embed-md "rdi-db-reqs.md" >}} @@ -68,23 +69,44 @@ file as described below. ### Using a private image registry -Add the RDI images from the -[download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-1.6.7.tgz) -to your local registry. -The example below shows how to specify the registry and image pull secret in the -[`values.yaml`](#the-valuesyaml-file) file for the Helm chart: +Add the RDI images from [dockerhub](https://hub.docker.com/u/redis) to your local registry. +You need the following RDI images: + +- [redis/rdi-api](https://hub.docker.com/r/redis/rdi-api) +- [redis/rdi-operator](https://hub.docker.com/r/redis/rdi-operator) +- [redis/rdi-monitor](https://hub.docker.com/r/redis/rdi-monitor) +- [redis/rdi-processor](https://hub.docker.com/r/redis/rdi-processor) +- [redis/rdi-collector-api](https://hub.docker.com/r/redis/rdi-collector-api) +- [redis/rdi-collector-initializer](https://hub.docker.com/r/redis/rdi-collector-initializer) + +In addition, the RDI Helm chart uses the following 3rd party images: + +- [redis/debezium-server](https://hub.docker.com/r/redis/debezium-server), originally `quay.io/debezium/server/3.0.8.Final`: + [Debezium](https://debezium.io/), an open source distributed platform for change data capture. +- [redis/reloader](https://hub.docker.com/r/redis/reloader), originally `ghcr.io/stakater/reloader:v1.1.0`: + [Reloader](https://github.com/stakater/Reloader), a K8s controller to watch changes to ConfigMaps + and Secrets and do rolling upgrades. +- [redis/kube-webhook-certgen](https://hub.docker.com/r/redis/kube-webhook-certgen), + originally `registry.k8s.io/ingress-nginx/kube-webhook-certgen/v20221220-controller-v1.5.1-58-g787ea74b6`: + [kube-webhook-certgen](https://github.com/jet/kube-webhook-certgen), K8s webhook certificate generator and patcher. + +The example below shows how to specify the registry and image pull secret in your +[`rdi-values.yaml`](#the-valuesyaml-file) file for the Helm chart: ```yaml global: - imagePullSecrets: [] - # - name: "image-pull-secret" - - image: - registry: docker.io - repository: redis + # Global image settings. + # If using a private image registry, update the default values accordingly. + image: + registry: your-registry + repository: your-repository # If different from "redis" + + # Image pull secrets to be used when using a private image registry. + imagePullSecrets: + - name: your-secret-name ``` -To pull images from a local registry, you must provide the image pull secret and in some cases also set the permissions. Follow the links below to learn how to use a private registry with: +To pull images from a private image registry, you must provide the image pull secret and in some cases also set the permissions. Follow the links below to learn how to use a private registry with: - [Rancher](https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-and-docker-registries#using-a-private-registry) - [OpenShift](https://docs.openshift.com/container-platform/4.17/openshift_images/managing_images/using-image-pull-secrets.html) @@ -102,34 +124,35 @@ To pull images from a local registry, you must provide the image pull secret and `rdi-values.yaml` file: ```bash - helm show values rdi-.tar.gz > rdi-values.yaml + helm show values rdi-.tar.gz > rdi-values.yaml ``` -1. Open the `rdi-values.yaml` file you just created and set the appropriate - values for your installation - (see [The `values.yaml` file](#the-valuesyaml-file) below for the full set of - available values). +1. Open the `rdi-values.yaml` file you just created, change or add the appropriate + values for your installation, and delete the values you have not changed to + use their default values. + See [The `values.yaml` file](#the-valuesyaml-file) for more details. -1. Start the installation: +1. Run the `helm upgrade --install` command: ```bash - helm install rdi rdi-.tar.gz -f rdi-values.yaml + helm upgrade --install rdi rdi-.tar.gz -f rdi-values.yaml -n rdi --create-namespace ``` - {{< note >}}By default, RDI will be installed in a namespace called - `rdi`. If you want to use a custom namespace, pass the option - `--namespace ` to the `helm install` command. + {{< note >}}The above command will install RDI in a namespace called + `rdi`. If you want to use a different namespace, pass the option + `-n ` to the `helm install` command instead. {{< /note >}} ### The `values.yaml` file -The annotated [`values.yaml`](https://helm.sh/docs/topics/charts/#templates-and-values) -file below describes the values you can set for the RDI Helm installation. +The [`values.yaml`](https://helm.sh/docs/topics/charts/#templates-and-values) file inside the +Helm chart contains the values you can set for the RDI Helm installation. +See the comments on each value for information on which values you may need to add or change +depending on your use case. -At a minimum, you must set the values of `RDI_REDIS_HOST` and `RDI_REDIS_PORT` -in the `global.rdiSysConfig` section and also `RDI_REDIS_PASSWORD` and -`JWT_SECRET_KEY` in `global.rdiSysSecret` to enable the basic connection to the -RDI database. RDI uses the value in `JWT_SECRET_KEY` to encrypt the +At a minimum, you must set the values of `connection.host`, `connection.port`, and `connection.password` +to enable the basic connection to the RDI database. +You must also set `api.jwtKey`, RDI uses this value to encrypt the [JSON web token (JWT)](https://jwt.io/) token used by RDI API. Best practice is to generate a value containing 32 random bytes of data (equivalent to 256 bits) and then encode this value as ASCII characters. Use the following @@ -140,354 +163,74 @@ command to generate the random key from the head -c 32 /dev/urandom | base64 ``` -{{< note >}}If you want to use -[Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}}) -to connect to your RDI deployment from outside the K8s cluster, you -must enable TLS. -{{< /note >}} - -The full `values.yaml` file is shown below: - -```yaml -# Default RDI values in YAML format. -# Variables to template configuration. - -global: - # Set this property when using a private image repository. - # Provide an array of image pull secrets. - # Example: - # imagePullSecrets: - # - name: pullSecret1 - # - name: pullSecret2 - imagePullSecrets: - - name: docker-config-jfrog - - # DO NOT modify this value. - vmMode: false - - # Indicates whether the deployment is intended for an OpenShift environment. - openShift: false - - image: - # Overrides the image tag for all RDI components. - # tag: 0.0.0 - - # If using a private repository, update the default values accordingly. - # Docker registry. - registry: docker.io - - # Docker image repository. - repository: redis - - # Configuration for the RDI ConfigMap. - rdiSysConfig: - # Log level for all RDI components. Valid options: DEBUG, INFO, ERROR. - # If specific component log levels are not set, this value will be used. - RDI_LOG_LEVEL: INFO - - # Log level for the RDI API. Valid options: DEBUG, INFO, ERROR. - # If not set, RDI_LOG_LEVEL will be used. - # RDI_LOG_LEVEL_API: INFO - - # Log level for the RDI Operator. Valid options: DEBUG, INFO, ERROR. - # If not set, RDI_LOG_LEVEL will be used. - # RDI_LOG_LEVEL_OPERATOR: INFO - - # Log level for the RDI processor. Valid options: DEBUG, INFO, ERROR. - # If not set, RDI_LOG_LEVEL will be used. - # RDI_LOG_LEVEL_PROCESSOR: INFO - - # Specifies whether the RDI is configured to use TLS. - RDI_REDIS_SSL: false - - # RDI_IMAGE_REPO: redis - - # This value must be set to the same tag as global.image.tag. - # RDI_IMAGE_TAG: "" - - # If using a private repository, set this value to the same secret name as in global.imagePullSecrets. - # RDI_IMAGE_PULL_SECRET: [] - - # The service IP of the RDI database. - # RDI_REDIS_HOST: "" - - # The port for the RDI database. - # RDI_REDIS_PORT: "" - - # Enable authentication for the RDI API. - # RDI_API_AUTH_ENABLED: "1" - - # Specifies whether the API Collector should be deployed. - # RDI_API_COLLECTOR_ENABLED: "0" - - # Configuration for the RDI Secret. - rdiSysSecret: - # Username and password for RDI database. - # If using the default password, keep the username as an empty string. - # RDI_REDIS_USERNAME: "" - # RDI_REDIS_PASSWORD: "" - - # Uncomment this property when using a TLS connection from RDI to its Redis database. - # DO NOT modify this value. - # RDI_REDIS_CACERT: /etc/certificates/rdi_db/cacert - - # Uncomment these properties when using an mTLS connection from RDI to its Redis database. - # DO NOT modify these values. - # RDI_REDIS_CERT: /etc/certificates/rdi_db/cert - # RDI_REDIS_KEY: /etc/certificates/rdi_db/key - - # The passphrase used to get the private key stored in the secret store when using mTLS. - # RDI_REDIS_KEY_PASSPHRASE: "" - - # The key used to encrypt the JWT token used by RDI API. Best practice is for this - # to contain 32 random bytes encoded as ASCII characters (equivalent to 256 bits of - # data). See `The values.yaml file` section above to learn how to generate the key. - # JWT_SECRET_KEY: "" - - rdiDbSSLSecret: - # Set to `true` when using a TLS connection from RDI to its Redis database. - enabled: false - - # The content of the CA certificate PEM file. - # Uncomment and set this property when using a TLS connection from RDI to its Redis database. - # cacert: "" - - # The content of the certificate PEM file. - # Uncomment and set this property when using an mTLS connection from RDI to its Redis database. - # cert: "" - - # The content of the private key PEM file. - # Uncomment and set this property when using an mTLS connection from RDI to its Redis database. - # key: "" - - # Container default security context. - # ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container - securityContext: - runAsNonRoot: true - runAsUser: 1000 - runAsGroup: 1000 - allowPrivilegeEscalation: false - -# Set `isOpenshift` to `true` if deploying on OpenShift. -reloader: - reloader: - isOpenshift: false - deployment: - containerSecurityContext: - allowPrivilegeEscalation: false - capabilities: - drop: - - ALL - securityContext: - runAsUser: null - -# Configuration of the RDI Operator. -operator: - image: - name: rdi-operator - - # Specify an imagePullPolicy. - # ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images - pullPolicy: IfNotPresent - - # Extra optional options for liveness probe. - # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ - liveness: - failureThreshold: 6 - periodSeconds: 10 - - # Extra optional options for readiness probe. - # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ - readiness: - failureThreshold: 6 - periodSeconds: 30 - - # Extra optional options for startup probe. - # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ - startup: - failureThreshold: 60 - periodSeconds: 5 - -fluentd: - image: - name: rdi-fluentd - - # Specify an imagePullPolicy. - # ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images - pullPolicy: IfNotPresent +If you use TLS to connect to the RDI database, you must set the +CA certificate content in `connection.ssl.cacert` (for TLS). In addition, if you +also use mTLS, you must set the client certificate and private key contents in +`connection.ssl.cert`, and `connection.ssl.key`. + +- You can add the certificate content directly in the `rdi-values.yaml` file + as follows: + + ```yaml + connection: + ssl: + enabled: true + cacert: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- + cert: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- + key: | + -----BEGIN PRIVATE KEY----- + ... + -----END PRIVATE KEY----- + ``` - rdiLogsHostPath: "/opt/rdi/logs" - podLogsHostPath: "/var/log/pods" - logrotateMinutes: "5" +- Alternatively, you can use the `--set-file` argument to set these values to + the content of your certificate files as follows: -rdiMetricsExporter: - image: - name: rdi-monitor - - # Specify an imagePullPolicy. - # ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images - pullPolicy: IfNotPresent - - # The RDI metrics service is set to ClusterIP, allowing access only from within the cluster. - # ref: http://kubernetes.io/docs/user-guide/services/ - service: - protocol: TCP - port: 9121 - targetPort: 9121 - type: ClusterIP - - # Configure extra options for liveness probe. - # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes - liveness: - failureThreshold: 6 - periodSeconds: 10 - - # Configure extra options for readiness probe. - # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes - readiness: - failureThreshold: 6 - periodSeconds: 30 - - # Configure extra options for startupProbe. - # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes - startup: - failureThreshold: 60 - periodSeconds: 5 - - # Configuration for the ServiceMonitor, which is used to scrape metrics from the RDI metrics service. - serviceMonitor: - # Set to `true` to activate the ServiceMonitor. - enabled: false - - # The endpoint from which Prometheus will scrape metrics. - path: /metrics - -# Configuration of the RDI API. -apiServer: - image: - name: rdi-api - - # Specify an imagePullPolicy. - # ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images - pullPolicy: IfNotPresent - - # The RDI API service is set to ClusterIP, allowing access only from within the cluster. - # ref: http://kubernetes.io/docs/user-guide/services/ - service: - type: ClusterIP - name: rdi-api - port: 8080 - targetPort: 8081 - - # Configure extra options for liveness probe. - # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes - liveness: - failureThreshold: 6 - periodSeconds: 10 - - # Configure extra options for readiness probe. - # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes - readiness: - failureThreshold: 6 - periodSeconds: 30 - - # Configure extra options for startupProbe. - # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes - startup: - failureThreshold: 60 - periodSeconds: 5 - -# Expose the RDI API service to be accessed from outside the cluster. -# ref: http://kubernetes.io/docs/user-guide/services/ -ingress: - # When `enabled` is set to `true`, RDI API Ingress will be created. - enabled: false - - # ref: https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress - # className: "" - - # Ingress resources configure routes based on the requested host. - # The optional Host field defines the hosts for routing. If omitted, it matches all hosts. - # Example: - # hosts: - # - example.com - # - another-example.com - - # Annotations to be added to the IngressClass resource. - # Example: - # annotations: - # kubernetes.io/ingress.class: "nginx" - # nginx.ingress.kubernetes.io/rewrite-target: / - - tls: - # Specifies whether the Ingress should be configured to use TLS. - enabled: false - - # When `enabled` is set to `true`, set this property to the content of the crt file. - # crt: "" - - # When `enabled` is set to `true`, set this property to the content of the key file. - # key: "" - -# When `openShift` is set to `true`, Route will be created automatically. -# Route exposes RDI API outside the cluster. -route: - tls: - # Specifies whether the Route should be configured to use TLS. - enabled: false - - # When `enabled` is set to `true`, set this property to the content of the crt file. - # crt: "" - - # When `enabled` is set to `true`, set this property to the content of the key file. - # key: "" - -collectorSourceMetricsExporter: - # The collector-source metrics service is set to ClusterIP, allowing access only from within the cluster. - # ref: http://kubernetes.io/docs/user-guide/services/ - service: - type: ClusterIP - port: 9092 - targetPort: 19000 - - # Configuration for the ServiceMonitor, which is used to scrape metrics from the collector-source metrics service. - serviceMonitor: - # Set to `true` to activate the ServiceMonitor. - enabled: false - - # The endpoint from which Prometheus will scrape metrics. - path: /metrics -``` + ```bash + helm upgrade --install rdi rdi-.tar.gz -f rdi-values.yaml -n rdi --create-namespace \ + --set connection.ssl.enabled=true \ + --set-file connection.ssl.cacert= \ + --set-file connection.ssl.cert= \ + --set-file connection.ssl.key= + ``` ## Check the installation To verify the status of the K8s deployment, run the following command: ```bash -helm list -n monitoring -n rdi +helm list -n rdi ``` -The output looks like the following. Check that `` is listed. +The output looks like the following. Check that the `rdi` release is listed. +With RDI 1.8.0 or later, check that the `default` release is also listed. ``` -NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION - rdi 1 2024-10-10 16:53... +0300 IDT deployed rdi-1.0.0 +NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION +default rdi 1 2025-05-08 ... deployed pipeline-0.1.0 +rdi rdi 3 2025-05-08 ... deployed rdi-1.0.0 ``` -Also, check that the following pods have `Running` status: +Also, check that all pods have `Running` status: ```bash kubectl get pod -n rdi -NAME READY STATUS RESTARTS AGE -rdi-api- 1/1 Running 0 29m -rdi-metric-l 1/1 Running 0 29m -rdi-operator- 1/1 Running 0 29m --reloader- 1/1 Running 0 29m -collector-api- 1/1 Running 0 29m +NAME READY STATUS RESTARTS AGE +collector-api- 1/1 Running 0 29m +rdi-api- 1/1 Running 0 29m +rdi-metric-exporter- 1/1 Running 0 29m +rdi-operator- 1/1 Running 0 29m +rdi-reloader- 1/1 Running 0 29m ``` -You can verify that the RDI API works by adding the server in +You can verify that the RDI API works by adding a connection to the RDI API server to [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}}). ## Using ingress controllers @@ -515,7 +258,7 @@ Specifically, ensure that one or both of the following Helm chart values is set: ## Prepare your source database -You must also configure your source database to use the CDC connector. See the +Before deploying a pipeline, you must configure your source database to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) section to learn how to do this. @@ -531,14 +274,18 @@ your pipeline. ## Uninstall RDI If you want to remove your RDI K8s installation, first run -the following commands. (If you installed with a custom namespace then +the following commands. (If you installed RDI into a custom namespace then replace `rdi` with the name of your namespace.) ```bash +kubectl delete pipeline default -n rdi helm uninstall rdi -n rdi kubectl delete namespace rdi ``` +{{< note >}}The line `kubectl delete pipeline default -n rdi` is only needed for RDI 1.8.0 or above. +{{< /note >}} + If you also want to delete the keys from your RDI database, connect to it with [`redis-cli`]({{< relref "/develop/tools/cli" >}}) and run a [`FLUSHALL`]({{< relref "/commands/flushall" >}}) command. diff --git a/content/integrate/redis-data-integration/installation/install-vm.md b/content/integrate/redis-data-integration/installation/install-vm.md index 3e5b0fefd..972ce808a 100644 --- a/content/integrate/redis-data-integration/installation/install-vm.md +++ b/content/integrate/redis-data-integration/installation/install-vm.md @@ -23,6 +23,14 @@ your source database. You can also {{< note >}}We recommend you always use the latest version, which is RDI {{< field "rdi_current_version" >}}. {{< /note >}} +## Create the RDI database + +RDI uses a database on your Redis Enterprise cluster to store its state +information. Use the Redis Enterprise Cluster Manager UI to create the RDI database with the following +requirements: + +{{< embed-md "rdi-db-reqs.md" >}} + ## Hardware sizing RDI is mainly CPU and network bound. @@ -30,10 +38,10 @@ Each of the RDI VMs should have at least: {{< embed-md "rdi-vm-reqs.md" >}} -## Install RDI on VMs +## VM Installation Requirements -You would normally install RDI on two VMs for high availability (HA) but you can also install -one just one VM if you don't need this. For example, you might not need HA during +You would normally install RDI on two VMs for High Availability (HA) but you can also install +just one VM if you don't need this. For example, you might not need HA during development and testing. {{< note >}}You can't install RDI on a host where a Redis Enterprise cluster @@ -53,8 +61,8 @@ You may encounter problems if you use `iptables` v1.6.1 and earlier in legacy mode with the following commands: ```bash -update-alternatives --set iptables /usr/sbin/iptables-legacy -update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy +sudo update-alternatives --set iptables /usr/sbin/iptables-legacy +sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy ``` Also, `iptables` versions 1.8.0-1.8.4 have known issues that can prevent RDI @@ -83,38 +91,25 @@ We recommend you turn off before installation using the command: ```bash -systemctl disable firewalld --now +sudo systemctl disable firewalld --now ``` However, if you do need to use `firewalld`, you must add the following rules: ```bash -firewall-cmd --permanent --add-port=6443/tcp #apiserver -firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16 #pods -firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16 #services -firewall-cmd --reload +sudo firewall-cmd --permanent --add-port=443/tcp # RDI API +sudo firewall-cmd --permanent --add-port=6443/tcp # kube-apiserver +sudo firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16 # Kubernetes pods +sudo firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16 # Kubernetes services +sudo firewall-cmd --reload ``` -You should also add [port rules](https://firewalld.org/documentation/howto/open-a-port-or-service.html) -for all the [RDI services]({{< relref "/integrate/redis-data-integration/reference/ports" >}}) -you intend to use: - -```bash -firewall-cmd --permanent --add-port=8080/tcp # (Required) rdi-operator/rdi-api -firewall-cmd --permanent --add-port=9090/tcp # vm-dis-reloader -firewall-cmd --permanent --add-port=9092/tcp # prometheus-service -firewall-cmd --permanent --add-port=9121/tcp # rdi-metric-exporter -``` - -{{}}You may also need to add similar rules to open other ports if your setup requires them. -{{}} - If you have `nm-cloud-setup.service` enabled, you must disable it and reboot the node with the following commands: ```bash -systemctl disable nm-cloud-setup.service nm-cloud-setup.timer -reboot +sudo systemctl disable nm-cloud-setup.service nm-cloud-setup.timer +sudo reboot ``` ### Ubuntu {#firewall-ubuntu} @@ -130,41 +125,35 @@ sudo ufw disable However, if you do need to use `ufw`, you must add the following rules: ```bash -ufw allow 6443/tcp #apiserver -ufw allow from 10.42.0.0/16 to any #pods -ufw allow from 10.43.0.0/16 to any #services +sudo ufw allow 443/tcp # RDI API +sudo ufw allow 6443/tcp # kube-apiserver +sudo ufw allow from 10.42.0.0/16 to any # Kubernetes pods +sudo ufw allow from 10.43.0.0/16 to any # Kubernetes services +sudo ufw reload ``` -You should also add [port rules](https://ubuntu.com/server/docs/firewalls) -for all the [RDI services]({{< relref "/integrate/redis-data-integration/reference/ports" >}}) -you intend to use: - -```bash -ufw allow 8080/tcp # (Required) rdi-operator/rdi-api -ufw allow 9090/tcp # vm-dis-reloader -ufw allow 9092/tcp # prometheus-service -ufw allow 9121/tcp # rdi-metric-exporter -``` - -{{}}You may also need to add similar rules to open other ports if your setup requires them. -{{}} - ## Installation steps Follow the steps below for each of your VMs: 1. Download the RDI installer from the - [Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-1.6.7.tar.gz) + [Redis download center](https://redis.io/downloads/) (from the *Modules, Tools & Integration* category) and extract it to your preferred installation folder. -1. Go to the installation folder: + ```bash + export RDI_VERSION= + wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-$RDI_VERSION.tar.gz + tar -xvf rdi-installation-$RDI_VERSION.tar.gz + ``` + +1. Go to the installation folder: ```bash cd rdi_install/$RDI_VERSION ``` -1. Run the installer as a privileged user: +1. Run the `install.sh` script as a privileged user: ```bash sudo ./install.sh @@ -175,23 +164,26 @@ Follow the steps below for each of your VMs: but this might be a problem if you have limited space in `/var` or your company policy forbids you to install there. You can select a different directory for the K3s installation using the - `--installation-dir` option with `install.sh` (or - [`redis-di install`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-install" >}})): + `--installation-dir` option with `install.sh`: ```bash - sudo ./install.sh --installation-dir + sudo ./install.sh --installation-dir ``` {{< /note >}} -RDI uses a database on your Redis Enterprise cluster to store its state -information. +The RDI installer collects all necessary configuration details and alerts you to potential issues, +offering options to abort, apply fixes, or provide additional information. +Once complete, it guides you through creating secrets and setting up your pipeline. -The installer gives you instructions to help you create secrets and create your pipeline. -It will ask you for cluster admin credentials during installation. +{{< note >}}It is strongly recommended to specify a hostname rather than an IP address for +connecting to your RDI database, for the following reasons: -Use the Redis console to create the RDI database with the following requirements: - -{{< embed-md "rdi-db-reqs.md" >}} +- Any DNS resolution issues will be detected during the installation rather than + later during pipeline deployment. +- If you use TLS, your RDI database CA certificate must contain the hostname you specified + either as a common name (CN) or as a subject alternative name (SAN). CA certificates + usually don't contain IP addresses. +{{< /note >}} {{< note >}}If you specify `localhost` as the address of the RDI database server during installation then the connection will fail if the actual IP address changes for the local @@ -207,7 +199,7 @@ sudo service k3s restart After the installation is finished, RDI is ready for use. -## Supply cloud DNS information +### Supply cloud DNS information {{< note >}}This section is only relevant if you are installing RDI on VMs in a cloud environment. @@ -217,9 +209,8 @@ If you are using [Amazon Route 53](https://aws.amazon.com/route53/), [Google Cloud DNS](https://cloud.google.com/dns?hl=en), or [Azure DNS](https://azure.microsoft.com/en-gb/products/dns) then you must supply the installer with the nameserver IP address -during installation (or with the `nameservers` property if you are -using [Silent installation](#silent-installation)). The table below -shows the appropriate IP address for each platform: +during installation. The table below +shows the appropriate IP address for each cloud provider: | Platform | Nameserver IP | | :-- | :-- | @@ -227,113 +218,40 @@ shows the appropriate IP address for each platform: | [Google Cloud DNS](https://cloud.google.com/dns?hl=en) | 169.254.169.254 | | [Azure DNS](https://azure.microsoft.com/en-gb/products/dns) | 168.63.129.16 | -If you are planning to use Route 53, you should first check that your VPC +If you are using Route 53, you should first check that your VPC is configured to allow it. See [DNS attributes in your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/AmazonDNS-concepts.html#vpc-dns-support) in the Amazon docs for more information. -## "Silent" installation - -You can use the -[installer script](#installation-steps) or the -[`redis-di install`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-install" >}}) -command with the `--file` option (or the shorter version `-f`) to supply answers -to the installer's questions automatically using properties from a -[TOML](https://toml.io/en/) file: - -```bash -./install.sh --file silent.toml -``` - -### Silent install example - -The following TOML file example shows the properties for a typical -silent install configuration: - -```toml -title = "RDI Silent Installer Config" - -scaffold = true -db_index = 4 -deploy_directory = "/opt/rdi/config" - -# Upstream DNS servers. This is needed if the installer detects a DNS resolver -# with a loopback address as an upstream DNS server. -# nameservers = ["8.8.8.8", "8.8.4.4"] - -# HTTPS port you want to expose the RDI API on, if different from 443. -# https_port = 5443 - -[rdi.database] -host = "localhost" -port = 12001 -username = "username" -password = "password" -ssl = true - -# Uncomment the properties in this section only if the RDI -# database uses TLS/mTLS. -# [rdi.database.certificates] -# ca = "/home/ubuntu/rdi/certs/ca.crt" -# cert = "/home/ubuntu/rdi/certs/client.crt" -# key = "/home/ubuntu/rdi/certs/client.key" -# passphrase = "foobar" -``` - -The sections below describe the properties in more detail. - -### Silent install properties - -#### Root - -| Property | Description | -|-- |-- | -| `title` | Text to identify the file. RDI doesn't use use this, so you can use any text you like. | -| `scaffold` | Do you want to enable [scaffolding]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-scaffold" >}}) during the install? (true/false) | -| `db_index` | Integer to specify the source database type for scaffolding. The options are 2 (MySQL/MariaDB), 3 (Oracle), 4 (PostgreSQL), and 5 (SQL Server). | -| `deploy_directory` | Path to the directory where you want to store the RDI configuration. | -| `nameservers` | Upstream DNS servers. This is needed if the installer detects a DNS resolver with a loopback address as an upstream DNS server (for example, `nameservers = ["8.8.8.8", "8.8.4.4"]`). | -| `https_port` | HTTPS port you want to expose the RDI API on, if different from 443. | - -#### `rdi.database` - -Use the properties in this section to specify your RDI database. - -| Property | Description | -|-- |-- | -| `host` | Hostname for the Redis database to store RDI state. | -| `port` | Port for the RDI database. | -| `username` | Username for the RDI database. | -| `password` | Password for the RDI database. | -| `ssl` | Is SSL enabled for the RDI database (true/false)? If this is false then RDI will ignore the settings in the [`rdi.database.certificates`](#rdidatabasecertificates) section. | +### Installing with High Availability -#### `rdi.database.certificates` +To install RDI with High Availability (HA), perform the [Installation steps](#installation-steps) +on two different VMs. The first VM will automatically become the active (primary) instance, +while the second VM will become the passive (secondary) one. +When starting the RDI installation on the second VM, the installer will detect that the RDI +database is already in use and ask you to confirm that you intend to install RDI with HA. -Use these properties only if the RDI database requires -[TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) or -[mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS). -You must also set `ssl` to `true` in the -[`rdi.database`](#rdidatabase) section to enable these properties. +After the installation is complete, you must set the source and target database secrets +on both VMs as described in [Deploy a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}). If you use `redis-di` to deploy your configuration, you only need to do this on one of the VMs, not both. -| Property | Description | -|-- |-- | -| `ca` | Path to the CA certificate file. | -| `cert` | Path to the client certificate file. | -| `key` | Path to the key file. | -| `passphrase` | Password for the private key (string). | +In a High Availability setup, the RDI pipeline is only active on the primary instance (VM). +The two RDI instances will use the RDI database for leader election. If the primary instance fails +to renew the lease in the RDI database, it will lose the leadership and a failover to the secondary instance +will take place. After the failover, the secondary instance will become the primary one, +and the RDI pipeline will be active on that VM. ## Prepare your source database -You must also configure your source database to use the CDC connector. See the +Before deploying a pipeline, you must configure your source database to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) section to learn how to do this. ## Deploy a pipeline When the installation is complete, and you have prepared the source database for CDC, -you are ready to start using RDI. See the guides to -[configuring]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines" >}}) and -[deploying]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) +you are ready to start using RDI. See the guides how to +[configure]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines" >}}) and +[deploy]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) RDI pipelines for more information. You can also configure and deploy a pipeline using [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}}). @@ -346,7 +264,7 @@ the uninstall script as a privileged user: sudo ./uninstall.sh ``` -The script will check you are sure before you proceed: +The script will ask if you are sure before proceeding: ``` This will uninstall RDI and its dependencies, are you sure? [y, N] diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 9159e3907..fb64c287a 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -22,18 +22,23 @@ Follow the steps below to upgrade an existing [VM installation]({{< relref "/integrate/redis-data-integration/installation/install-vm" >}}) of RDI: -1. Download the RDI installer from the - [Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-1.6.7.tar.gz) +1. Download the RDI installer from the [Redis download center](https://redis.io/downloads/) (in the *Modules, Tools & Integration* category) and extract it to your preferred installation folder. + ```bash + export RDI_VERSION= + wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-$RDI_VERSION.tar.gz + tar -xvf rdi-installation-$RDI_VERSION.tar.gz + ``` + 1. Go to the installation folder: ```bash cd rdi_install/$RDI_VERSION ``` -1. Run the `upgrade` script as a privileged user. Note that you must pass +1. Run the `upgrade.sh` script as a privileged user. Note that you must pass your RDI password to the script unless the password is empty. ```bash @@ -43,7 +48,7 @@ of RDI: ### Recovering from failure during a VM upgrade If the previous version is v1.4.4 or later, go to the `rdi_install/` -directory and run `sudo ./upgrade.sh`, as described in the section +directory and run `sudo ./upgrade.sh` to revert to the that version, as described in the section [Upgrading a VM installation](#upgrading-a-vm-installation) above. If the version you are replacing is earlier than v1.4.4, follow these steps: @@ -66,18 +71,16 @@ If the version you are replacing is earlier than v1.4.4, follow these steps: sudo redis-di upgrade --rdi-host --rdi-port ``` -{{< note >}}If the `collector-source` or the `processor` are not in the `Running` state during +{{< note >}}If the `collector-source` or the `processor` pods are not in the `Running` state after the upgrade, you must run `redis-di deploy` and check again that they are both in the `Running` state. {{< /note >}} -### Upgrading a VM installation with High availability +### Upgrading a VM installation with High Availability -If there is an active pipeline, the upgrade process will involve upgrading RDI on the active -VM first which will cause downtime for the collector-source (see -[Upgrade a VM installation](#upgrade-a-vm-installation) above). Afterwards, the passive -VM will be upgraded. Switching over won't eliminate the downtime because switching between -VMs also requires a about a minute of downtime. +If there is an active pipeline, upgrade RDI on the active VM first. +This will cause a short pipeline downtime of up to two minutes. +Afterwards, upgrade RDI on the passive VM. This will not cause any downtime. ## Upgrading a Kubernetes installation @@ -85,74 +88,72 @@ Follow the steps below to upgrade an existing [Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}}) installation of RDI: -1. Download the new versions of the images, if you are using a private registry: +1. If you are using a private registry, pull the new versions of all images listed in + [Using a private image registry]({{< relref "/integrate/redis-data-integration/installation/install-k8s#using-a-private-image-registry" >}}) + and add them to your local registry. - ```bash - docker pull redis/rdi-processor:tagname - docker pull redis/rdi-operator:tagname - docker pull redis/rdi-api:tagname - docker pull redis/rdi-monitor:tagname - docker pull redis/rdi-collector-initializer - docker pull redis/rdi-collector-api - ``` +1. Download the RDI Helm chart tar file from the [Redis download center](https://redis.io/downloads/) + (in the *Modules, Tools & Integration* category). -1. Download the RDI helm chart tar file from the - [Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-1.6.7.tgz). +1. Adapt your `rdi-values.yaml` file to any changes in the new RDI version if needed. + See also [Upgrading to RDI 1.8.0 or later from an earlier version](#upgrading-to-rdi-180-or-later-from-an-earlier-version). + Before making any changes, save your existing `rdi-values.yaml` if you need to revert + to the old RDI version for any reason. 1. Run the `helm upgrade` command: ```bash - helm upgrade [RELEASE_NAME] [CHART] + helm upgrade --install rdi rdi-.tar.gz -f rdi-values.yaml -n rdi ``` - Note that you don't need to - [deploy]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) - again after this step. - -### Verifying the upgrade +Note that you don't need to +[deploy]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) +the RDI configuration again after this step. -Check the upgrade with the following command: +### Upgrading to RDI 1.8.0 or later from an earlier version -```bash - sudo k3s kubectl get all -n -``` +When upgrading to RDI 1.8.0 or later from an earlier version +you must adapt your `rdi-values.yaml` file to the following changes: -You should find that all the pods are running (they will have `1/1` in the `READY` column of the -command's output). -Check for any pods that don't have `1/1` in the `READY` column (which is the second -column). For example, the pod below has `0/1` in the second column, which indicates the -deployment hasn't worked: +- All collector and processor values that were previously under `collector`, + `collectorSourceMetricsExporter`, and `processor` have been moved to + `operator.dataPlane.collector` and `operator.dataPlane.processor`. +- `global.collectorApiEnabled` has been moved to `operator.dataPlane.collectorApi.enabled`, + and is now a boolean value, not `"0"` or `"1"`. +- `api.authEnabled` is also now a boolean value, not `"0"` or `"1"`. +- The following values have been deprecated: `rdiMetricsExporter.service.protocol`, + `rdiMetricsExporter.service.port`, `rdiMetricsExporter.serviceMonitor.path`, + `api.service.name`. -```bash - 0/1 CrashLoopBackOff 1881 (91s ago) 6d17h -``` +### Verifying the upgrade -You can also check that the latest version is running using the following command on one of -the pods: +Check that all pods have `Running` status: ```bash -sudo k3s kubectl describe -n +kubectl get all -n rdi ``` -Search for the image tag `Image: docker.io/redis/:` -in the command's output to verify the version. - -If you find that the upgrade hasn't worked for any reason, then run the `helm upgrade` -command again (as described in the section +If you find that the upgrade did not work as expected for any reason, +then run the `helm upgrade` command again (as described in the section [Upgrading a Kubernetes installation](#upgrading-a-kubernetes-installation) above), -but this time with the previous version you were upgrading from. This will restore your -previous working state. +but this time with the previous version you were upgrading from, and using +your saved `rdi-values.yaml` for that version. This will restore your previous working state. + +{{< note >}}Downgrading from RDI 1.8.0 or later to an earlier version using `helm upgrade` +will not work. If you need to perform such an upgrade, uninstall RDI completely first as +described in [Uninstall RDI]({{< relref "/integrate/redis-data-integration/installation/install-k8s#uninstall-rdi" >}}), +and then install the old version. +{{< /note >}} ## What happens during the upgrade? -The upgrade process replaces the current RDI components with the new versions: +The upgrade process replaces the current RDI components with their new versions: - Firstly, the control plane components are replaced. At this point, the pipeline is still active but monitoring will be disconnected. - -- Secondly, the pipeline data path components are replaced with the new versions. - If a pipeline is active while upgrading, the `collector-source` will be restarted - as a result of restarting the `collector-initializer`. The pipeline will pause for - about two minutes but it will catch up very quickly after restarting. - The pipeline data and state are both stored in Redis, so data will never normally - be lost during the downtime while upgrading. +- Secondly, the pipeline data plane components are replaced. + If a pipeline is active while upgrading, the `collector-source` and `processor` + pods will be restarted. The pipeline will pause for up to two minutes but it + will catch up very quickly after restarting. + The pipeline data and state are both stored in Redis, so data will not + be lost during the upgrade. diff --git a/content/integrate/redis-data-integration/quick-start-guide.md b/content/integrate/redis-data-integration/quick-start-guide.md index 225437e4f..dc4ecfe73 100644 --- a/content/integrate/redis-data-integration/quick-start-guide.md +++ b/content/integrate/redis-data-integration/quick-start-guide.md @@ -50,10 +50,10 @@ RDI will create the pipeline template for your chosen source database type at (see [Prepare the pipeline](#prepare-the-pipeline) below). At the end of the installation, RDI CLI will prompt you to set the access secrets -for both the source PostgreSQL database and the Redis RDI database. RDI needs these to +for both the source PostgreSQL database and the target Redis database. RDI needs these to run the pipeline. -Use the Redis console to create the RDI database with the following requirements: +Use the Redis Enterprise Cluster Manager UI to create the RDI database with the following requirements: {{< embed-md "rdi-db-reqs.md" >}} @@ -90,7 +90,7 @@ To create a context, use the command: ```bash -redis-di add-context --rdi-host --rdi-port --cluster-host --cluster-api-port --cluster-user +redis-di add-context --rdi-host --rdi-port ``` These options are required but there are also a few others you can save, such as TLS credentials, if @@ -113,7 +113,7 @@ contexts. You can use [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}}) to deploy the pipeline by adding a connection to the RDI API -endpoint (which has the same IP address as your RDI VM and uses port 8083) and then clicking the **Deploy** button. You can also deploy it with the following command: +endpoint (which has the same hostname or IP address as your RDI VM and uses the default HTTPS port 443) and then clicking the **Deploy** button. You can also deploy it with the following command: ```bash redis-di deploy --dir @@ -148,4 +148,4 @@ To see the RDI pipeline working in CDC mode: - Run [`redis-di status --live`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-status" >}}) to see the flow of records. -- User [Redis Insight]({{< relref "/develop/tools/insight" >}}) to look at the data in the target database. +- Use [Redis Insight]({{< relref "/develop/tools/insight" >}}) to look at the data in the target database. From 46ab546d56777f91862902c780833963a6412684 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Fri, 9 May 2025 14:30:45 +0300 Subject: [PATCH 2/7] Add wget command for the RDI Helm chart --- .../redis-data-integration/installation/install-k8s.md | 8 ++++++++ .../redis-data-integration/installation/upgrade.md | 5 +++++ 2 files changed, 13 insertions(+) diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index 76a3a387f..964f23ae7 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -47,12 +47,20 @@ or from your own [private image registry](#using-a-private-image-registry). Complete the following steps before installing the RDI Helm chart: - [Create the RDI database](#create-the-rdi-database) on your Redis Enterprise cluster. + - Create a [user]({{< relref "/operate/rs/security/access-control/create-users" >}}) for the RDI database if you prefer not to use the default password (see [Access control]({{< relref "/operate/rs/security/access-control" >}}) for more information). + - Download the RDI Helm chart tar file from the [Redis download center](https://redis.io/downloads/) (in the *Modules, Tools & Integration* category) . + + ```bash + export RDI_VERSION= + wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-$RDI_VERSION.tgz + ``` + - If you want to use a private image registry, [prepare it with the RDI images](#using-a-private-image-registry). diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index fb64c287a..d3a9067ba 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -95,6 +95,11 @@ installation of RDI: 1. Download the RDI Helm chart tar file from the [Redis download center](https://redis.io/downloads/) (in the *Modules, Tools & Integration* category). + ```bash + export RDI_VERSION= + wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-$RDI_VERSION.tgz + ``` + 1. Adapt your `rdi-values.yaml` file to any changes in the new RDI version if needed. See also [Upgrading to RDI 1.8.0 or later from an earlier version](#upgrading-to-rdi-180-or-later-from-an-earlier-version). Before making any changes, save your existing `rdi-values.yaml` if you need to revert From f5408cc380db8802451ecb5d57688b849bea9168 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Fri, 9 May 2025 14:40:47 +0300 Subject: [PATCH 3/7] Update debezium-server image --- .../redis-data-integration/installation/install-k8s.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index 964f23ae7..d69960a1c 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -89,7 +89,7 @@ You need the following RDI images: In addition, the RDI Helm chart uses the following 3rd party images: -- [redis/debezium-server](https://hub.docker.com/r/redis/debezium-server), originally `quay.io/debezium/server/3.0.8.Final`: +- [redislabs/debezium-server](https://hub.docker.com/r/redislabs/debezium-server), originally `quay.io/debezium/server/3.0.8.Final`: [Debezium](https://debezium.io/), an open source distributed platform for change data capture. - [redis/reloader](https://hub.docker.com/r/redis/reloader), originally `ghcr.io/stakater/reloader:v1.1.0`: [Reloader](https://github.com/stakater/Reloader), a K8s controller to watch changes to ConfigMaps From 14d3162bf422d19d47ea5a3bee0919be3d9861c2 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Fri, 9 May 2025 18:39:01 +0300 Subject: [PATCH 4/7] Update content/integrate/redis-data-integration/installation/install-vm.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../integrate/redis-data-integration/installation/install-vm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/installation/install-vm.md b/content/integrate/redis-data-integration/installation/install-vm.md index 972ce808a..5a13f3334 100644 --- a/content/integrate/redis-data-integration/installation/install-vm.md +++ b/content/integrate/redis-data-integration/installation/install-vm.md @@ -249,7 +249,7 @@ section to learn how to do this. ## Deploy a pipeline When the installation is complete, and you have prepared the source database for CDC, -you are ready to start using RDI. See the guides how to +you are ready to start using RDI. See the guides on how to [configure]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines" >}}) and [deploy]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) RDI pipelines for more information. You can also configure and deploy a pipeline From 8c71c4fb3111b3801cb6c326c16535ed3ba4b726 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Fri, 9 May 2025 18:39:19 +0300 Subject: [PATCH 5/7] Update content/integrate/redis-data-integration/installation/upgrade.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../integrate/redis-data-integration/installation/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index d3a9067ba..6df919480 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -48,7 +48,7 @@ of RDI: ### Recovering from failure during a VM upgrade If the previous version is v1.4.4 or later, go to the `rdi_install/` -directory and run `sudo ./upgrade.sh` to revert to the that version, as described in the section +directory and run `sudo ./upgrade.sh` to revert to that version, as described in the section [Upgrading a VM installation](#upgrading-a-vm-installation) above. If the version you are replacing is earlier than v1.4.4, follow these steps: From f986f3a44e70059e7b9aa124cfc8ebfc3e29fd07 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Fri, 9 May 2025 18:44:37 +0300 Subject: [PATCH 6/7] Address comments --- .../redis-data-integration/installation/install-k8s.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index d69960a1c..9e6848da4 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -155,7 +155,7 @@ To pull images from a private image registry, you must provide the image pull se The [`values.yaml`](https://helm.sh/docs/topics/charts/#templates-and-values) file inside the Helm chart contains the values you can set for the RDI Helm installation. -See the comments on each value for information on which values you may need to add or change +See the comments by each value for more information about the values you may need to add or change depending on your use case. At a minimum, you must set the values of `connection.host`, `connection.port`, and `connection.password` From 501486c8d33d2c45375a5cdc89d3d1d48b0097d7 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Fri, 9 May 2025 18:50:38 +0300 Subject: [PATCH 7/7] Add information about RDI image tags. --- .../redis-data-integration/installation/install-k8s.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index 9e6848da4..821780706 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -78,7 +78,7 @@ file as described below. ### Using a private image registry Add the RDI images from [dockerhub](https://hub.docker.com/u/redis) to your local registry. -You need the following RDI images: +You need the following RDI images with tags matching the RDI version you want to install: - [redis/rdi-api](https://hub.docker.com/r/redis/rdi-api) - [redis/rdi-operator](https://hub.docker.com/r/redis/rdi-operator) @@ -89,12 +89,13 @@ You need the following RDI images: In addition, the RDI Helm chart uses the following 3rd party images: -- [redislabs/debezium-server](https://hub.docker.com/r/redislabs/debezium-server), originally `quay.io/debezium/server/3.0.8.Final`: +- [redislabs/debezium-server:3.0.8.Final-rdi.1](https://hub.docker.com/r/redislabs/debezium-server), + based on `quay.io/debezium/server/3.0.8.Final` with minor modifications: [Debezium](https://debezium.io/), an open source distributed platform for change data capture. -- [redis/reloader](https://hub.docker.com/r/redis/reloader), originally `ghcr.io/stakater/reloader:v1.1.0`: +- [redis/reloader:v1.1.0](https://hub.docker.com/r/redis/reloader), originally `ghcr.io/stakater/reloader:v1.1.0`: [Reloader](https://github.com/stakater/Reloader), a K8s controller to watch changes to ConfigMaps and Secrets and do rolling upgrades. -- [redis/kube-webhook-certgen](https://hub.docker.com/r/redis/kube-webhook-certgen), +- [redis/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6](https://hub.docker.com/r/redis/kube-webhook-certgen), originally `registry.k8s.io/ingress-nginx/kube-webhook-certgen/v20221220-controller-v1.5.1-58-g787ea74b6`: [kube-webhook-certgen](https://github.com/jet/kube-webhook-certgen), K8s webhook certificate generator and patcher.