feat(/config-migration): add migration service to convert legacy configuration to the new format#641
feat(/config-migration): add migration service to convert legacy configuration to the new format#641
Conversation
Deployment results
Logs #24004982267 |
| private readonly bucket: string | ||
|
|
||
| constructor(config: S3Config) { | ||
| this.bucket = config.bucket |
There was a problem hiding this comment.
The reason I didn't add this AWS module globally in the root was because the migration only uses it for now.
Swapping the aws library now also requires bucket and region. we can hardcode or add as dev vars
There was a problem hiding this comment.
if the structure overall is good, we can extract in smaller PRs
pnpm-workspace.yaml
Outdated
| - 'components' | ||
| - 'shared/*' | ||
| - 'localenv/*' | ||
| - 'migrations/*' |
There was a problem hiding this comment.
These and related dependencies will be installed in CI, which won't really be used. Any way we can ignore them in CI, and only install when needed locally?
There was a problem hiding this comment.
sure. introduced different workspace for migrations, but will have to cd into /migrations, manually install and run tests.
| }) | ||
| } | ||
|
|
||
| export async function dryRun( |
There was a problem hiding this comment.
Having a separate function/login as dryRun won't really be a dry-run. Please add only one function, and instead pass it a mock-s3 so it acts like dry-run (for non-recoverable actions), while keeping a single flow.
| await s3.putJson(walletAddress, newData) | ||
| await s3.deleteFromLegacy(walletAddress) |
There was a problem hiding this comment.
What if either of these fails? Can we have something like a transaction here?
There was a problem hiding this comment.
removed delete.
we'll manually delete everything after we make sure everything went smoothly. manually or in batch.
| } | ||
| const newData = convertToConfiguration(legacyData, walletAddress) | ||
| await s3.putJson(walletAddress, newData) | ||
| await s3.deleteFromLegacy(walletAddress) |
There was a problem hiding this comment.
Can we batch delete in end? We should write a local log file to write which files (and/or a cursor) have already been processed, then delete them in end when we're sure.
There was a problem hiding this comment.
we'll manually delete everything after we make sure everything went smoothly. manually or in batch.
| } | ||
|
|
||
| // only run main CLI if this file executed directly | ||
| if (import.meta.url === pathToFileURL(process.argv[1]).href) { |
2 participants
Summary
one-time CLI to migrate legacy S3 config data to the new format
migrationsCLI that reads wallet configs from the legacy S3 prefix, converts them to the new shape, writes them to the new prefix.Adds migrations/s3/ as a standalone S3 client package wrapping client-s3, shared by the CLI.
Includes a dry-run mode (--dry-run) that simulates all S3 writes/deletes without touching real data, and writes its own separate log file.
Each run produces a timestamped log file under migrations recording successful, skipped, and failed wallets.
Migration packages are excluded from the root pnpm workspace — they live under their own migrations/pnpm-workspace.yaml and are installed separately (cd migrations && pnpm install), so CI is not affected