feat!(website, backend): Update bulk file submission/revision format to use metadata file by tombch · Pull Request #6922 · loculus-project/loculus

tombch · 2026-07-17T18:48:36Z

PR description and docs still need updating with further details

Summary

Restructures how files are submitted in bulk submissions/revisions, as well as how file mappings are provided from the website to the backend for submissions/revisions - previously, bulk and form website submission/revisions attached a separate file mapping object. Now, file mappings for each submission are written to the metadata file by the website before sending for submission/revision.

New files.<category> columns are introduced within the metadata, and uploaded files must be linked to within here - this can be done simply by adding a space separated list of: the file names (name1 name2 name3, if uploading a flat folder of files) or the file paths ( name1::path1 name2::path2 name3::path3, if uploading a more complicated folder structure, to allow for different submissions to use the same file name within a bulk upload).

Screenshot

Subfolders can be submitted (paths show in download info) using fileName::filePath in metadata file:

Files referenced in metadata but not uploaded are added in a warning and files that are not referenced in the metadata but are uploaded are also warned on

PR Checklist

All necessary documentation has been adapted.
The implemented feature is covered by appropriate, automated tests.
Any manual testing that has been done is documented (i.e. what exactly was tested?)

🚀 Preview: https://bulk-file-submissions.loculus.org

…file category json if they exist, and centralised file validation after creation in aux table

…xisting tests working

…ly, with file paths for linking to metadata columns

…pplying of file maps to metadata tsv

claude · 2026-07-17T18:49:40Z

This PR may be related to: #6883, #5820

…dividual submissions, and updated website tests

…culus into bulk-file-submissions

… different failed file linkages (missing from upload, metadata etc) and reflected the results of this in the UI for bulk submissions. removed submissionFileMapping linked/unlinked status as no longer needed

…ed/restructured FolderUploadComponent tests

…apping

tombch · 2026-07-28T10:09:03Z

+                headers.push(sourceHeader);
+                indices.push(sourceIndex);
+            }
+        });


Another thing to note - the files category columns currently appear as unmapped in the column mapper - they should probably be recognised as input fields similar to fastaIds field

…rd filed

… adding, replacing and discarding files

…bmissions

claude · 2026-07-28T15:32:12Z

Claude finished @tombch's task in 5m 41s —— View job

Claude finished reviewing

Summary

This is a well-structured refactor that moves bulk file linkage from a separate JSON mapping into files.<category> columns embedded in the metadata TSV. The new fileMapping.ts linkage model (linked/reused/missing/orphaned/shadowed) is a nice improvement over the previous folder-only matching, and it's backed by strong test coverage on both sides (fileMapping.spec.ts, MetadataEntryTest.kt, and the expanded file-sharing.spec.ts integration tests covering reuse/replace/discard/add flows and blocking-error scenarios).

Backend parsing (extractAndValidateFileIdAndName in MetadataEntry.kt) correctly uses lastIndexOf for the name:id split — safe since : is already a forbidden filename character (FileMappingPreconditionValidator.validateFilename), so there's no ambiguity even for filenames that happen to contain the separator character before rejection. The old-JSON-mapping backward-compat path (submissionParams.files merged via mergeFileCategories, metadata columns taking priority) and the file-group-ownership re-validation against the merged data in SubmitModel.kt also look correct.

Two minor issues left as inline comments:

GetSubmittedDataHelpers.kt:28 — hardcodes the : separator instead of reusing the already-defined FILE_NAME_ID_SEPARATOR constant, which could drift from the parse path over time.
DataUploadForm.tsx:147 — recomputes getFileLinkage(...) and shadows the fileLinkage already produced by the useMemo above, which covers the same condition. Minor redundancy/readability nit, not a correctness bug.

I also independently confirmed the concern raised in your own review comment on ColumnMapping.ts:115: applyTo force-includes files.* columns regardless of what the mapping UI shows, so those columns can look "unmapped" in the mapper while still being passed through — worth addressing for UI clarity, but not a functional bug since the data does flow through correctly.

No security, performance, or documentation concerns beyond what's noted above. Didn't run the test suites myself (no Docker/gradle execution in this session) — the existing coverage looks thorough for the new logic.

…from bulk revisions

anna-parker · 2026-07-29T21:36:45Z

-                    validateSubmissionIdSetsForFiles(metadataSubmissionIds, fileSubmissionIds)
-                    validateFileGroupOwnership(submittedFiles, submissionParams, uploadId)
+            submissionMetrics.timeWritePhase(endpoint, organism, VALIDATE_FILE_MAPPING_PHASE) {
+                // File mappings in submissionParams can contain submission Ids not present in the metadata


Ah do I understand correctly that the backend actually still supports the old submission format and will accept having the files in a fileMapping JSON?

Sounds like a good rollout strategy! It might be good to still have a follow up PR where the JSON is completely removed to ensure the removal will be smooth - I started doing that locally now and does seem like it will still be a bit of a change!

Yeah, currently I haven't removed any of the separate file mapping stuff from the backend, but for single and bulk submission/revision the website now sends the file mapping within the metadata tsv. The submit-edited-data endpoint didn't have to change how it handled the file mapping as that one is completely JSON for everything

anna-parker · 2026-07-30T14:45:22Z

Sign in to view

+    }
+
+    // TODO: Update when moving away from UUIDs to more user-friendly file IDs
+    val fileId = try {


we discussed adding a specific length check for now to improve the error message in case part of the UUID is accidentally deleted

but maybe it's quick enough it doesn't matter :)

anna-parker · 2026-07-30T14:47:42Z

+            submissionMetrics.timeWritePhase(endpoint, organism, VALIDATE_FILE_MAPPING_PHASE) {
+                // File mappings in submissionParams can contain submission Ids not present in the metadata
+                // This is implicitly validated for the file mappings within the metadata column
+                // TODO: This can be removed once file mappings JSON support is removed


The backend tests actually all still rely on the fact that the fileMapping can be passed in submission and do not check that the files will also be parsed from the metadata. I guess this PR is for an intermittent state where both submission options work but might still make sense to update at least some of the tests - I ended up doing that in #7012 (when the fileMapping JSON is completely removed)

…opefully more readable with comments

anna-parker · 2026-07-30T16:09:26Z

@tombch and I discussed some blockers for merging this PR:

Docs that need updating: https://loculus.org/for-users/submit-extra-files/, also backend endpoints in the FilesController need updating (ah claude actually did this for me in 5c693fb)
Adding a migration guide for users using the current feature in the PR description
Ensure files.rawReads shows as mapped to a keyword in the column mapping:

anna-parker · 2026-07-30T17:00:55Z

+            const text = columnMapping
+                ? await (await columnMapping.applyTo(metadataFile)).text()
+                : await metadataFile.text();
+            if (!controller.signal.aborted) {


its probably nicer to use if (controller.signal.aborted) return; to have less indented code

anna-parker · 2026-07-30T17:04:26Z

    const [columnMapping, setColumnMapping] = useState<ColumnMapping | null>(null);

+    useEffect(() => {
+        const controller = new AbortController();


Im not sure we actually need a controller I think just a boolean called cancelled would be enough (it needs to be set to true in the cleanup function)

tombch added 9 commits July 14, 2026 15:00

Add files.<fileCategory> column parsing in metadataEntryStreamAsSequence

dd2f5a4

Incorporate file categories from columns into submission, overriding …

7934bef

…file category json if they exist, and centralised file validation after creation in aux table

Moved filemapping json precondition validators back for now to keep e…

fdd5eee

…xisting tests working

Added tests for metadata entry file column parsing

9621030

Added filename-fileid to downloaded submission data

e4bb833

Updating FolderUploadComponent to store flat arrays of files internal…

f6ef4a2

…ly, with file paths for linking to metadata columns

added visual linkage by path between metadata tsv and files

c68591c

Cleaned up mapping functions and added error handling

d03dcdf

Moved new file mapping logic into fileMapping.ts, added merging and a…

2898644

…pplying of file maps to metadata tsv

claude Bot added website Tasks related to the web application backend related to the loculus backend component labels Jul 17, 2026

tombch changed the title ~~feat!(website): Updated bulk file submission/revision format to use metadata file~~ feat!(website): Update bulk file submission/revision format to use metadata file Jul 17, 2026

tombch changed the title ~~feat!(website): Update bulk file submission/revision format to use metadata file~~ feat!(website, backend): Update bulk file submission/revision format to use metadata file Jul 17, 2026

tombch and others added 13 commits July 20, 2026 17:36

Merge branch 'main' into bulk-file-submissions

5fc3153

Merge branch 'main' into bulk-file-submissions

bc5202d

Consistent file keying by path, reimplemented previous uploads for in…

c098cba

…dividual submissions, and updated website tests

Merge branch 'main' into bulk-file-submissions

ec01a61

Fixing existing integration tests

eda9c48

Merge branch 'bulk-file-submissions' of github.com:loculus-project/lo…

5085dd8

…culus into bulk-file-submissions

Fixing existing integration tests

0203055

Cleanup

fd8f11f

Added getFileLinkage function - used in submit for error handling for…

86ad6f5

… different failed file linkages (missing from upload, metadata etc) and reflected the results of this in the UI for bulk submissions. removed submissionFileMapping linked/unlinked status as no longer needed

Removed dead formSubmissionId prop

5b27014

Merge branch 'main' into bulk-file-submissions

a7c7c72

Updated extractFileIdAndName to use variable for separator

c164f23

De-brittle metadata entry tests

54cf4ba

tombch added the preview Triggers a deployment to argocd label Jul 27, 2026

tombch added 2 commits July 27, 2026 14:41

Clean up folder upload component

9abecf8

Removed redundant dotfile check on handleAddAdditionalFiles and updat…

4045a9d

…ed/restructured FolderUploadComponent tests

tombch commented Jul 28, 2026

View reviewed changes

Comment thread integration-tests/tests/specs/features/file-sharing.spec.ts Outdated

tombch added 4 commits July 28, 2026 11:18

cleanup

d52174b

filemapping tests

9b02690

Gate handleSubmit on successful metadata file parsing of submission m…

fd01e7b

…apping

cleanup

9f60fe1

tombch commented Jul 28, 2026

View reviewed changes

tombch added 3 commits July 28, 2026 14:55

Integration tests for EditPage component reuse, update, add and disca…

9a4f753

…rd filed

Added tests for bulk submission/revision blocking errors and reusing,…

a72568c

… adding, replacing and discarding files

Fixing integration tests and testing use of same file names across su…

c7e3540

…bmissions

tombch marked this pull request as ready for review July 28, 2026 15:31

claude Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread backend/src/main/kotlin/org/loculus/backend/utils/GetSubmittedDataHelpers.kt Outdated

claude Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread website/src/components/Submission/DataUploadForm.tsx Outdated

tombch added 5 commits July 29, 2026 13:36

add titles to file paths in FolderUploadComponent

effbc9f

Use variable for name-id separator

94bc2f6

updated wording

0bad387

Fixing integration tests

ef1720c

Merge remote-tracking branch 'origin/main' into bulk-file-submissions

1dce8bc

tombch removed the preview Triggers a deployment to argocd label Jul 29, 2026

tombch and others added 2 commits July 29, 2026 16:59

Fixing integration test race condition accessing resulting version 2 …

cc280dc

…from bulk revisions

Merge branch 'main' into bulk-file-submissions

4b538da

anna-parker reviewed Jul 29, 2026

View reviewed changes

tombch added the preview Triggers a deployment to argocd label Jul 30, 2026

anna-parker mentioned this pull request Jul 30, 2026

Bulk submission cleanup #7012

Draft

3 tasks

anna-parker reviewed Jul 30, 2026

View reviewed changes

Updated file mapping with better validation, consolidated types and h…

3583a6e

…opefully more readable with comments

anna-parker reviewed Jul 30, 2026

View reviewed changes

Uh oh!

Conversation

tombch commented Jul 17, 2026 • edited by anna-parker Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Screenshot

PR Checklist

Uh oh!

claude Bot commented Jul 17, 2026

Uh oh!

Uh oh!

tombch Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Claude finished reviewing

Uh oh!

Uh oh!

Uh oh!

anna-parker Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

tombch Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anna-parker Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as low quality.

Uh oh!

theosanderson Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

anna-parker Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anna-parker commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anna-parker Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

anna-parker Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tombch commented Jul 17, 2026 •

edited by anna-parker

Loading

claude Bot commented Jul 28, 2026 •

edited

Loading

tombch Jul 30, 2026 •

edited

Loading

anna-parker Jul 30, 2026 •

edited

Loading

anna-parker commented Jul 30, 2026 •

edited

Loading