A much simpler migration script that relies on spacepy

scripts/migrate/README.md

@@ -1,26 +1,25 @@
-# Science-file S3/DB migration
+# Science-file S3 migration
 
 One-off tooling to re-version IMAP science files: it rewrites each file's
-name (and the embedded CDF metadata) to the new versioning scheme and updates
-the matching `science_files` rows in the RDS database.
+name (and the embedded CDF metadata) to the new versioning scheme.
 
 The heavy lifting runs on a short-lived EC2 instance so it has in-region, egress-free
-access to the S3 bucket and the RDS instance. The `run` script provisions that
-instance, ships the migration code to it, executes it, and tears everything down.
+access to the S3 buckets. The `run` script provisions that instance, ships the
+migration code to it, executes it, and tears everything down.
 
 ```
 run (your laptop)
  ├─ ensures IAM role + instance profile + key pair exist
- ├─ launches (or reuses) a t3.large EC2 instance
- ├─ opens the RDS security group to that instance's IP
- ├─ scp run_remote.sh + migrate.py to the instance
- └─ ssh: run_remote.sh
-         ├─ installs uv + Python deps
-         └─ python migrate.py   <- the actual S3 / DB migration
+ ├─ ensures an SSH security group (port 22) exists
+ ├─ launches (or reuses) an m9g.48xlarge EC2 instance
+ ├─ scp run_remote_rename.sh + rename.py to the instance
+ └─ ssh: run_remote_rename.sh
+         ├─ builds the NASA CDF C library + installs uv + Python deps
+         └─ python rename.py   <- the actual S3 migration
 ```
 
-On exit (success, failure, or Ctrl-C) the instance is terminated and the
-security-group rule is revoked automatically.
+On exit (success, failure, or Ctrl-C) the instance is terminated automatically.
+The SSH security group is created once and left in place between runs.
 
 ---
 
@@ -31,8 +30,8 @@ On the **local** machine that runs `run`:
 - AWS CLI v2, authenticated for the target account. `run` uses
   `AWS_PROFILE=imap-dev` — set up that profile (`aws configure --profile imap-dev`)
   or edit the variable at the top of `run`.
-- Permission to manage IAM roles/instance profiles, EC2 key pairs and
-  instances, and to modify the RDS security group.
+- Permission to manage IAM roles/instance profiles, EC2 key pairs,
+  security groups, and instances.
 - `ssh`, `scp`, `openssl`, `envsubst` on `PATH`.
 
 Run `bash run` to test out the setup. This won't actually run the migration, simply
@@ -46,40 +45,35 @@ Configure the run by editing the variables near the top of `run`, then execute i
 
 | Variable | Default | Meaning                                                    |
 |----------|---------|------------------------------------------------------------|
-| `COPY_FILES` | `0`     | `1` = copy/rewrite files in S3 (stage 1).                  |
-| `MODIFY_ROWS` | `0`     | `1` = update `file_path` in the database (stage 2).         |
+| `DRY_RUN`   | `1`     | `1` = only print the old -> new name mapping, write nothing. Set to `0` to actually migrate. |
 | `OVERWRITE` | `0`     | `1` = re-copy destination files even if they already exist. |
 | `MAX_FILES` | `1000`  | Max CDF/PKTS files to process this run (`0` = all).        |
 
-> **`COPY_FILES` and `MODIFY_ROWS` are mutually exclusive.** `migrate.py`
-> asserts you never enable both in the same run — do it in stages (below).
+`SRC_PREFIX`, `DRY_RUN`, and `MAX_FILES` can also be overridden from the
+environment without editing `run`, e.g. `DRY_RUN=0 MAX_FILES=500 bash run`.
+This is handy for running several prefixes in parallel (see below).
 
 ---
 
 ## Workflow
 
-### Incremental file copying (stage 1)
+### Incremental file copying
 
-```bash
-COPY_FILES=1
-MODIFY_ROWS=0
-```
-
-Reads each source object under `imap/`, rewrites the CDF metadata
+Reads each source object under `$SRC_PREFIX` in `$SRC_BUCKET`, rewrites the CDF metadata
 (`Data_version`, `Logical_file_id`, `Parents`) and writes it to the new name
-under the `renamed/` prefix. PKTS files are copied as-is to the new name.
-Existing objects under `renamed/` are skipped, so this stage is resumable and
-can be run in batches (see below).
+under the same `$SRC_PREFIX`, but now in `$DST_BUCKET`. PKTS files are copied as-is to
+the new name. Existing objects at destination are skipped, so this stage is resumable
+and can be run in batches (see below).
 
 `MAX_FILES` limits how many CDF/PKTS files a single run processes. Because
-stage 1 **skips destinations that already exist** under `renamed/`, repeated
+it **skips destinations that already exist**, repeated
 runs with the same `MAX_FILES` walk through the whole set a batch at a time:
 
 ```bash
-# in run: COPY_FILES=1, MODIFY_ROWS=0, OVERWRITE=0, MAX_FILES=1000
+# in run: DRY_RUN=0, OVERWRITE=0, MAX_FILES=1000
 bash run   # copies the first 1000 not-yet-copied files
 bash run   # copies the next 1000
-bash run   # ... repeat until everything is under renamed/
+bash run   # ... repeat until everything is copied to DST_BUCKET
 ```
 
 Set `MAX_FILES=0` to process everything in one run. *Not recommended.* except for
@@ -89,45 +83,36 @@ To **regenerate** files you have already copied (e.g. after fixing the metadata
 logic), set `OVERWRITE=1`. This disables the skip-if-exists behavior and
 re-copies the selected files in place.
 
-Copying runs across multiple CPU cores automatically (one worker per core).
-`run` uses `t3.large` (2 vCPUs), so two files are rewritten in parallel;
-use a larger instance type if you want more throughput. `t3.xlarge` has 4 vCPUs.
-`t3.2xlarge` has 8 vCPUs.
-
-
-### Manual step - promote the renamed files
-
-After verifying `renamed/`, back up the existing `imap/` tree and then bulk-move
-objects from `renamed/..` to their final `imap/..` paths. `run` does not do this move
-for you.
-
-You will want to keep the `ancillary/`, `dependency/` and `spice/` trees in `imap/`
-intact, since these do not have rows in the science_files table and are not affected by
-the renaming.
-
-### Update the database (stage 2)
-
-```bash
-COPY_FILES=0
-MODIFY_ROWS=1
-```
+This is single threaded since I'm having trouble getting `spacepy` to behave in a
+multi-processing environment. However, a bigger instance will still make it go faster.
 
-Updates each `science_files.file_path` from the old name to the new one.
+### Running several prefixes in parallel
 
-Then run it:
+To speed things up you can shard the work by `SRC_PREFIX` and run several copies
+of `run` at once, one per prefix. `SRC_PREFIX`, `DRY_RUN`, and `MAX_FILES` can
+be overridden from the environment, so **do not edit `run` in place** for this —
+pass them on the command line instead:
 
 ```bash
-bash run        # or ./run
+DRY_RUN=0 SRC_PREFIX=imap/lo/   bash run     # terminal 1
+DRY_RUN=0 SRC_PREFIX=imap/mag/  bash run     # terminal 2
+DRY_RUN=0 SRC_PREFIX=imap/swe/  bash run     # terminal 3
 ```
 
-#### Tips
+Note the default is `DRY_RUN=1`, which only prints the `old -> new` mapping and
+writes **nothing** — pass `DRY_RUN=0` to actually migrate.
 
-- Inspect s3 bucket in a separate terminal to monitor file copy progress.
+Each copy derives its own EC2 instance name from the prefix
+(`INSTANCE_TAG=s3-transition-<slug>`), so every terminal gets its **own**
+instance, its own `$HOME` on that instance, and a teardown that only kills its
+own box — the runs don't interfere. The shared IAM role, key pair, and security
+group are created-if-missing, so a quick warm-up `bash run` before fanning out
+avoids a first-run creation race on those.
 
-  ```bash
-  aws s3 ls s3://sds-data-593025701104/renamed/ --recursive --summarize --human-readable
-  ```
+Caveats:
 
-- Set `INTERACTIVE=1` to provision the instance and drop into an SSH shell
-instead of running the migration. The instance is still torn down when you exit
-the shell.
+- `SRC_PREFIX` **must end with `/`.
+- Give every terminal a **different** `SRC_PREFIX`. Two runs with the same
+  prefix resolve to the same instance name and will collide.
+- Each shard launches its own `m9g.48xlarge`, so N terminals = N instances
+  running at once — watch the cost and any vCPU quota.

scripts/migrate/ec2-perms.json

@@ -6,22 +6,24 @@
       "Action": [
         "s3:ListBucket"
       ],
-      "Resource": "arn:aws:s3:::${AWS_BUCKET}"
+      "Resource": [
+        "arn:aws:s3:::${SRC_BUCKET}",
+        "arn:aws:s3:::${DST_BUCKET}"
+      ]
     },
     {
       "Effect": "Allow",
       "Action": [
-        "s3:GetObject",
-        "s3:PutObject"
+        "s3:GetObject"
       ],
-      "Resource": "arn:aws:s3:::${AWS_BUCKET}/*"
+      "Resource": "arn:aws:s3:::${SRC_BUCKET}/*"
     },
     {
       "Effect": "Allow",
       "Action": [
-        "secretsmanager:GetSecretValue"
+        "s3:PutObject"
       ],
-      "Resource": "arn:aws:secretsmanager:${AWS_DEFAULT_REGION}:${AWS_ACCOUNT}:secret:sdp-database-cred*"
+      "Resource": "arn:aws:s3:::${DST_BUCKET}/*"
     }
   ]
 }