A much simpler migration script that relies on spacepy

[vineetbansal]

Closes #1414

Similar in operational details as a previous PR, but now we simply open the cdf through spacepy, modify Data_version, Logical_file_id, and Parents, and writes it out using spacepy. Similar to what Menlo did on their end.

There's no multiprocessing/multihreading in the script (spacepy is crashing on multiprocessing), but we can still partition the runs by SRC_PREFIX (the paths relative to the bucket root that we want to process).

We should probably do this in parallel for each instrument, something like:

terminal 1:
DRY_RUN=0 MAX_FILES=0 SRC_PREFIX="old_version/imap/lo" bash run

terminal 2:
DRY_RUN=0 MAX_FILES=0 SRC_PREFIX="old_version/imap/mag" bash run

[pleasant-menlo]

Per Tim's comment, it would be nice to update the File_naming_convention attribute to source_descriptor_datatype_yyyyMMdd_vMMM.mmmm as well.

[tmplummer]

I pointed out on the ticket for this that at least one attribute is missing: File_naming_convention: source_descriptor_datatype_yyyyMMdd_vNNN. That is from the global cdf attributes yaml. I don't know if anyone has scrubbed products for other possible global attributes that need to be updated.

[vineetbansal]

Ah - got it now. That file looked like a readme of sorts so I missed the fact that its an actual attribute mapping yaml. Done.

[jtniehof]

I did a quick check and didn't see any other missing globals.

[tech3371]

I haven't looked at code details. I have couple general questions.

1. Are we doing file rename all over again with this new script? or list of files that failed from last transition?
2. I wonder if the source of some files renaming failing could be we used cdflib only to rename L1-L3 files. Should we rename L1-L2 files using cdflib and L3 files using spacepy like they normally do in their batch job? The reason I say this is because I don't know what impact reading and writing files using different library and impact of that in the future. Eg. was L3 cdf load with spacepy failing because we used cdflib during this transition, which could add additional or less information and vice versa. I hope this is not an issue since both are writing CDF file.

[tech3371]

Can you overwrite previous script to reduce code to review and to see what we are doing differently this time? or comment which ones we have changed? This will help me a lot since lot of context switching is happening.

[vineetbansal]

@tech3371 - the problem with using cdflib is outlined in all the issues that Menlo opened up - the one line takeaway is that by using write_cdf, we're modifying the CDF is more ways than just the 3 attributes we wish to change, and also possibly modifying the filepath itself (i.e. multiple source files with different vXXX versions map to the same vNNN.nnnn version, which is why some of the supposedly generated files are missing, and some of the generated files are corruputed.

Unless we're willing to take the time to figure out which files are affected and which are okay, the simpler option is to pretend that the previous renaming never happened and do it again using this script, and overwrite the previously generated vNNN.nnnn files (which may be corrupted, or have data different from their vXXX counterparts) with the ones generated using this script.

[vineetbansal]

@tech3371 - I don't think code redundancy should be a priority right now. We can do that once we're past this release. I'd keep the old scripts for reference. The conceptual difference between the old approach and the new approach are:

migrate (take 1) is the DB-driven, single-bucket, multiprocess, two-stage (S3 + database) approach built on the our own imap_processing/imap_data_accessstack.migrate_take2is a simpler, regex/S3-key-driven, bucket-to-bucket, single-threaded, S3-only rename built on rawspacepy`.

All the bash scripts are driving scripts - the function you want to read is def upgrade_cdf.

At the end of the day all we're doing is executing a function (upgrade_cdf) on all files in an S3 bucket. Everything else is fluff. If you can think of a simpler way to do it that's great too!

[jtniehof]

Note from the meeting today, check for accuracy: because some of the filenames changed in the old script (the date, not just the version), we will need to compare the newly-renamed files against the currently-renamed files in production, and see where there are differences. Any files that are in production but not in the new set of files will need to be removed from the bucket and the index (after the renaming, the reindexer was run and would have ingested those as "new" files).

[tmplummer]

They python script looks good to me with one suggested change. It would be good to get @bryan-harter to review the AWS related files.

[tmplummer]

Should the initial migrate code be removed as part of this PR?

[vineetbansal]

Yes - @tech3371 mentioned this earlier. I'll merge everything into the migrate folder right now (also so the person doing the ancillary migration is not confused). EDIT: done.

[vineetbansal]

hmm - I had no idea ancillaries were not getting renamed. I guess no one else here did either.

I think it might be good to make this change (though I don't understand why this regex matches science files only and excludes ancillary files - someone else will need to confirm that).

Bryan will have to decide whether to leave mag science files as-is or re-rerun the renaming on them after making this fix, or if fixing those can be a task folded in with the exercise of fixing the Parents in the mag ancillary files themselves.

[pleasant-menlo]

Good catch. On our side, we forgot to think about the possibility that ancillary files would have the .cdf extension, since our L3 files are all different formats.

Ancillary files have a different number of underscore-delimited sections in their filename because they do not have the data level section.

[sapols]

@vineetbansal The regex separates them structurally. Science names carry the data level as its own underscore-delimited token (imap_<instrument>_<level>_<descriptor>_<date>_vNNN.cdf), which is what _l\d[a-z]?_ matches. Ancillary names have no level token; the descriptor sits in that position, and even the level-ish MAG descriptors are hyphen-joined, so the match fails: in imap_mag_l1b-calibration_20240229_v001.cdf the regex requires _ after l1b but finds -. The MAG offsets files additionally have two date fields (_20260101_20260101_) where science names have one. (.pkts matching is deliberate; L0 files were renamed by the migration too.)

Also verified empirically, not just by inspection: I ran this exact gate over the Parents of all 577 MAG ancillary files and 283 migrated science files sampled across every instrument and level—it matched every old-scheme science name and passed every ancillary name through untouched.

[vineetbansal]

makes sense..you might want to walk to Bryan's office to discuss whether he's ok rerunning this for mag after merge since he may not be following this thread closely and we're out of time.

[sapols]

I work from home unfortunately. @bryan-harter please weigh in here?

[bryan-harter]

I see this now! I can work on this in a bit.

[bryan-harter]

Ok, I believe I fixed them all. Would you mind trying again?

[sapols]

@bryan-harter I wasn't sure if you were talking to me or Vineet, but I went ahead and had Fable 5 re-run the audit script from my comment in imap_processing#3348 and here's what it came back with:

Confirmed — the full audit was re-run fresh (no cached downloads): the 124 L2 mode-days that had mangled ancillary parents are all clean now, as are the migrated L1B files, and nothing else changed — no new damage anywhere in the sweep.

Status of what remains, in the terms of that same comment:

• 265 L2 mode-days with stale old-scheme L1C parents (population 1 in the Task 2 audit) — unchanged, pending the rewrite-vs-reprocess decision after the ancillary fix.
• 298 stale ancillary offsets files (Task 1) — unchanged on prod; the fix script is ready whenever.

[sapols]

@vineetbansal are you planning on incorporating my suggested fix to your rename.py?

[vineetbansal]

@sapols - I incorporated your suggested fix (through the UI) but it seems partial - _OLD_SCIENCE_RE is not defined anywhere. I guess Fable was expecting me to do that part.

And now the whole suggestion has disappeared from github anyway. Do you mind doing your AI magic to suggest the fix? I'm kinda new to this seemingly partial agentic integration in github.

EDIT: I see it now. I'll make the change.

[sapols]

Ah sorry for the confusion there @vineetbansal. Fable helped write the regex but it was my call to use GitHub's suggestion feature that way. Totally see how splitting the code between suggestion and comment wasn't obvious. Glad you figured it out 😅 Looks good now

[bryan-harter]

Just a general comment, we have now already used this script to alter/rename all of the files that we need to, and I don't think it is super likely that we will need this script again in the future. Are there thoughts about keeping it in the repo v.s. not keeping it? I don't mind either way. But perhaps there are reasons to keep the repo "clean" (reducing AI agent context windows?)

[bryan-harter]

I'll also approve it for now, and feel free to merge it so we can keep our open PRs down! It shouldn't really affect things either way.