|
| 1 | +========================================================= |
| 2 | +Migrate from DynamoDB to ScyllaDB Alternator Using Docker |
| 3 | +========================================================= |
| 4 | + |
| 5 | +In this tutorial, you will replicate 1,000,000 items from a DynamoDB table to ScyllaDB Alternator. |
| 6 | + |
| 7 | +All the scripts and configuration files shown on the tutorial can be found in our `GitHub repository <https://github.com/scylladb/scylla-migrator/tree/master/docs/source/tutorials/dynamodb-to-scylladb-alternator>`_. |
| 8 | + |
| 9 | +The whole system is composed of the DynamoDB service, a Spark cluster with a single worker node, and a ScyllaDB cluster with a single node, as illustrated below: |
| 10 | + |
| 11 | +.. image:: architecture.png |
| 12 | + :alt: Architecture |
| 13 | + :width: 600 |
| 14 | + |
| 15 | +To follow this tutorial, you need to install `Docker <https://docker.com>`_ and the `AWS CLI <https://aws.amazon.com/cli/>`_. |
| 16 | + |
| 17 | +---------------------------------------------------- |
| 18 | +Set Up the Services and Populate the Source Database |
| 19 | +---------------------------------------------------- |
| 20 | + |
| 21 | +In an empty directory, create the following ``docker-compose.yaml`` file to define all the services: |
| 22 | + |
| 23 | +.. literalinclude:: docker-compose.yaml |
| 24 | + :language: YAML |
| 25 | + |
| 26 | +Let’s break down this Docker Compose file. |
| 27 | + |
| 28 | +1. We define the DynamoDB service by reusing the official image ``amazon/dynamodb-local``. We use the TCP port 8000 for communicating with DynamoDB. |
| 29 | +2. We define the Spark master and Spark worker services by using a custom image (see below). Indeed, the official Docker images for Spark 3.5.1 only support Scala 2.12 for now, but we need Scala 2.13. We mount the local directory ``./spark-data`` to the Spark master container path ``/app`` so that we can supply the Migrator jar and configuration to the Spark master node. We expose the ports 8080 and 4040 of the master node to access the Spark UIs from our host environment. We allocate 2 cores and 4 GB of memory to the Spark worker node. As a general rule, we recommend allocating 2 GB of memory per core on each worker. |
| 30 | +3. We define the ScyllaDB service by reusing the official image ``scylladb/scylla``. We use the TCP port 8001 for communicating with ScyllaDB Alternator. |
| 31 | + |
| 32 | +Create the ``Dockerfile`` required by the Spark services at path ``./dockerfiles/spark/Dockerfile`` and write the following content: |
| 33 | + |
| 34 | +.. literalinclude:: dockerfiles/spark/Dockerfile |
| 35 | + :language: Dockerfile |
| 36 | + |
| 37 | +This ``Dockerfile`` installs Java and a Spark distribution. It uses a custom shell script as entry point. Create the file ``./dockerfiles/spark/entrypoint.sh``, and write the following content: |
| 38 | + |
| 39 | +.. literalinclude:: dockerfiles/spark/entrypoint.sh |
| 40 | + :language: sh |
| 41 | + |
| 42 | +The entry point takes an argument that can be either ``master`` or ``worker`` to control whether to start a master node or a worker node. |
| 43 | + |
| 44 | +Prepare your system for building the Spark Docker image with the following commands, which create the ``spark-data`` directory and make the entry point executable: |
| 45 | + |
| 46 | +.. code-block:: sh |
| 47 | +
|
| 48 | + mkdir spark-data |
| 49 | + chmod +x entrypoint.sh |
| 50 | +
|
| 51 | +Finally, start all the services with the following command: |
| 52 | + |
| 53 | +.. code-block:: sh |
| 54 | +
|
| 55 | + docker compose up |
| 56 | +
|
| 57 | +Your system's Docker daemon will download the DynamoDB and ScyllaDB images and build your Spark Docker image. |
| 58 | + |
| 59 | +Check that you can access the Spark cluster UI by opening http://localhost:8080 in your browser. You should see your worker node in the workers list. |
| 60 | + |
| 61 | +.. image:: spark-cluster.png |
| 62 | + :alt: Spark UI listing the worker node |
| 63 | + :width: 883 |
| 64 | + |
| 65 | +Once all the services are up, you can access your local DynamoDB instance and your local ScyllaDB instance by using the standard AWS CLI. Make sure to configure the AWS CLI as follows before running the ``dynamodb`` commands: |
| 66 | + |
| 67 | +.. code-block:: sh |
| 68 | +
|
| 69 | + # Set dummy region and credentials |
| 70 | + aws configure set region us-west-1 |
| 71 | + aws configure set aws_access_key_id dummy |
| 72 | + aws configure set aws_secret_access_key dummy |
| 73 | + # Access DynamoDB |
| 74 | + aws --endpoint-url http://localhost:8000 dynamodb list-tables |
| 75 | + # Access ScyllaDB Alternator |
| 76 | + aws --endpoint-url http://localhost:8001 dynamodb list-tables |
| 77 | +
|
| 78 | +The last preparatory step consists of creating a table in DynamoDB and filling it with random data. Create a file named ``create-data.sh``, make it executable, and write the following content into it: |
| 79 | + |
| 80 | +.. literalinclude:: create-data.sh |
| 81 | + :language: sh |
| 82 | + |
| 83 | +This script creates a table named ``Example`` and adds 1 million items to it. It does so by invoking another script, ``create-25-items.sh``, which uses the ``batch-write-item`` command to insert 25 items in a single call: |
| 84 | + |
| 85 | +.. literalinclude:: create-25-items.sh |
| 86 | + :language: sh |
| 87 | + |
| 88 | +Every added item contains an id and five columns, all filled with random data. |
| 89 | +Run the script ``./create-data.sh`` and wait for a couple of hours until all the data is inserted (or change the last line of ``create-data.sh`` to insert fewer items). |
| 90 | + |
| 91 | +--------------------- |
| 92 | +Perform the Migration |
| 93 | +--------------------- |
| 94 | + |
| 95 | +Once you have set up the services and populated the source database, you are ready to perform the migration. |
| 96 | + |
| 97 | +Download the latest stable release of the Migrator in the ``spark-data`` directory: |
| 98 | + |
| 99 | +.. code-block:: |
| 100 | +
|
| 101 | + wget https://github.com/scylladb/scylla-migrator/releases/latest/download/scylla-migrator-assembly.jar \ |
| 102 | + --directory-prefix=./spark-data |
| 103 | +
|
| 104 | +Create a configuration file in ``./spark-data/config.yaml`` and write the following content: |
| 105 | + |
| 106 | +.. literalinclude:: spark-data/config.yaml |
| 107 | + :language: YAML |
| 108 | + |
| 109 | +This configuration tells the Migrator to read the items from the table ``Example`` in the ``dynamodb`` service, and to write them to the table of the same name in the ``scylla`` service. |
| 110 | + |
| 111 | +Finally, start the migration with the following command: |
| 112 | + |
| 113 | +.. literalinclude:: run-migrator.sh |
| 114 | + :language: sh |
| 115 | + |
| 116 | +This command calls ``spark-submit`` in the ``spark-master`` service with the file ``scylla-migrator-assembly.jar``, which bundles the Migrator and all its dependencies. |
| 117 | + |
| 118 | +In the ``spark-submit`` command invocation, we explicitly tell Spark to use 4 GB of memory; otherwise, it would default to 1 GB only. We also explicitly tell Spark to use 2 cores. This is not really necessary as the default behavior is to use all the available cores, but we set it for the sake of illustration. If the Spark worker node had 20 cores, it would be better to use only 10 cores per executor to optimize the throughput (big executors require more memory management operations, which decrease the overall application performance). We would achieve this by passing ``--executor-cores 10``, and the Spark engine would allocate two executors for our application to fully utilize the resources of the worker node. |
| 119 | + |
| 120 | +The migration process inspects the source table, replicates its schema to the target database if it does not exist, and then migrates the data. The data migration uses the Hadoop framework under the hood to leverage the Spark cluster resources. The migration process breaks down the data to transfer chunks of about 128 MB each, and processes all the partitions in parallel. Since the source is a DynamoDB table in our example, each partition translates into a `scan segment <https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.ParallelScan>`_ to maximize the parallelism level when reading the data. Here is a diagram that illustrates the migration process: |
| 121 | + |
| 122 | +.. image:: process.png |
| 123 | + :alt: Migration process |
| 124 | + :width: 700 |
| 125 | + |
| 126 | +During the execution of the command, a lot of logs are printed, mostly related to Spark scheduling. Still, you should be able to spot the following relevant lines: |
| 127 | + |
| 128 | +.. code-block:: text |
| 129 | +
|
| 130 | + 24/07/22 15:46:13 INFO migrator: ScyllaDB Migrator 0.9.2 |
| 131 | + … |
| 132 | + 24/07/22 15:46:20 INFO alternator: We need to transfer: 2 partitions in total |
| 133 | + 24/07/22 15:46:20 INFO alternator: Starting write… |
| 134 | + 24/07/22 15:46:20 INFO DynamoUtils: Checking for table existence at destination |
| 135 | +
|
| 136 | +And when the migration ends, you will see the following line printed: |
| 137 | + |
| 138 | +.. code-block:: text |
| 139 | +
|
| 140 | + 24/07/22 15:46:24 INFO alternator: Done transferring table snapshot |
| 141 | +
|
| 142 | +During the migration, it is possible to monitor the underlying Spark job by opening the Spark UI available at http://localhost:4040. |
| 143 | + |
| 144 | +.. image:: stages.png |
| 145 | + :alt: Spark stages |
| 146 | + :width: 900 |
| 147 | + |
| 148 | +`Example of a migration broken down in 6 tasks. The Spark UI allows us to follow the overall progress, and it can also show specific metrics such as the memory consumption of an executor`. |
| 149 | + |
| 150 | +In our example the size of the source table is ~200 MB. In practice, it is common to migrate tables containing several terabytes of data. If necessary, and as long as your DynamoDB source supports a higher read throughput level, you can increase the migration throughput by adding more Spark worker nodes. The Spark engine will automatically spread the workload between all the worker nodes. |
| 151 | + |
0 commit comments