[FIX] sync curation extraction by jdkent · Pull Request #1248 · neurostuff/neurostore

jdkent · 2025-12-06T19:30:51Z

honestly a long standing issue that nick has brought up before, this make the alignment between stubs and studies within studysets direct so people do not lose their annotations (and sometimes analyses) when they update their meta-analyses. (updating meta-analyses is something we want to support, if we want to do "living meta-analyses").

Copilot

Pull request overview

This PR adds support for tracking curation stub UUIDs on studyset-study associations, enabling better synchronization between curation and extraction workflows. The changes allow the system to maintain a stable mapping between curated stubs and their corresponding study versions, even when studies are updated or re-ingested.

Key changes:

Added curation_stub_uuid column to the studyset_studies association table with appropriate indexing and uniqueness constraints
Updated schema serialization/deserialization to handle stub UUID mappings in studyset payloads
Modified frontend helper functions to preserve stub-to-study mappings when syncing curation data to extraction studysets

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
store/backend/neurostore/models/data.py	Adds `curation_stub_uuid` column to StudysetStudy model with relationship configurations
store/backend/migrations/versions/8e3f3d8a9b5b_add_curation_stub_uuid.py	Database migration to add the new column, index, and unique constraint
store/backend/neurostore/schemas/data.py	Updates schema serialization to capture and normalize stub UUID mappings
store/backend/neurostore/resources/data.py	Implements logic to update stub mappings when creating/updating studysets
store/backend/neurostore/resources/base.py	Refines permission checks for nested record updates
store/backend/neurostore/tests/api/test_studysets.py	Adds test coverage for stub UUID capture and persistence
store/scripts/backfill_curation_stub_uuid.py	Provides a backfill script for existing data
compose/neurosynth-frontend/src/helpers/Extraction.helpers.tsx	Adds `mapStubsToStudysetPayload` function to maintain stub-to-study mappings
compose/neurosynth-frontend/src/helpers/Extraction.helpers.spec.ts	Unit tests for the new helper function
compose/neurosynth-frontend/src/pages/Project/components/MoveToExtractionDialog.tsx	Updates to use new stub mapping helper
compose/neurosynth-frontend/src/pages/Extraction/components/ExtractionOutOfSync.tsx	Updates sync logic to preserve stub mappings
cypress/e2e/workflows/ingestion/Ingestion.cy.tsx	Minor test fixture path corrections

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

nicoalee · 2025-12-10T22:09:31Z

dependent on:
neurostuff/neurostore-spec#98

nicoalee

Pushed minor changes removing an unused function argument.
Overall it looks good and it works on my end.

We just need to merge in and rebuild the typescript SDKs to add the curation_stub_map property in the studyset update object

I tested:

creating and then ingesting two stubs without doi/pmid/pmcid creates two separate studies/base studies. Previous annotations are retained.
creating and then ingesting two stubs with the same doi/pmid/pmcid creates two separate studies/base studies. Previous annotations are retained.

jdkent · 2025-12-11T16:44:13Z

looking at the implementation, I do not want curation_stub_map to be a part of the public api, just adds more opportunities for there being 2 sources of truth. curation_stub_uuid should be passed with id within the study object with the parameter studies. Sorry for putting you on a path to try to make that schema work.

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

Copilot · 2025-12-11T16:48:29Z

    const queryClient = useQueryClient();
    return useMutation<AxiosResponse<StudysetReturn>, AxiosError, StudysetRequest, unknown>(
-        (studyset) => API.NeurostoreServices.StudySetsService.studysetsPost(studyset),
+        (studyset) => API.NeurostoreServices.StudySetsService.studysetsPost(undefined, undefined, undefined, studyset),


Three undefined parameters suggest the API signature may have changed. Consider using named parameters or an options object to make the call site more maintainable and less error-prone.

Suggested change

(studyset) => API.NeurostoreServices.StudySetsService.studysetsPost(undefined, undefined, undefined, studyset),

(studyset) => API.NeurostoreServices.StudySetsService.studysetsPost({ studyset }),

jdkent · 2025-12-12T18:55:30Z

I'll merge this by EOD!

nicoalee · 2025-12-12T19:09:58Z

looking at the implementation, I do not want curation_stub_map to be a part of the public api, just adds more opportunities for there being 2 sources of truth. curation_stub_uuid should be passed with id within the study object with the parameter studies. Sorry for putting you on a path to try to make that schema work.

no worries! i was actually wondering why we had the two sources of truth (curation_stub_map and the studyset.studyset_studies) so this makes a lot more sense

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ub_uuid.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

jdkent requested a review from Copilot December 6, 2025 23:16

Copilot AI reviewed Dec 6, 2025

View reviewed changes

Comment thread store/backend/neurostore/resources/data.py Outdated

Comment thread store/backend/neurostore/schemas/data.py Outdated

Comment thread store/backend/neurostore/resources/base.py Outdated

Comment thread compose/neurosynth-frontend/src/helpers/Extraction.helpers.tsx Outdated

jdkent requested review from Copilot and nicoalee December 7, 2025 05:26

Copilot AI reviewed Dec 7, 2025

View reviewed changes

Comment thread store/backend/neurostore/resources/data.py Outdated

Comment thread store/backend/neurostore/schemas/data.py Outdated

Comment thread compose/neurosynth-frontend/src/pages/Extraction/components/ExtractionOutOfSync.tsx Outdated

jdkent mentioned this pull request Dec 7, 2025

check if cached annotation/studyset is different from uploaded one. #458

Closed

jdkent commented Dec 7, 2025

View reviewed changes

Comment thread store/backend/migrations/versions/8e3f3d8a9b5b_add_curation_stub_uuid.py Outdated

nicoalee reviewed Dec 11, 2025

View reviewed changes

jdkent requested a review from Copilot December 11, 2025 16:45

Copilot AI reviewed Dec 11, 2025

View reviewed changes

nicoalee approved these changes Dec 12, 2025

View reviewed changes

jdkent added 15 commits December 12, 2025 17:18

track curation_uuid in studyset_study

33bc2e7

modify frontend to accomodate exporting curation_uuid

7c19ff1

make sure stub follows user specified changes

76a41db

update openapi

8f05eb4

fix tests in neurostore

2d25d31

fix revision

e1dd796

fix ingestion test

c668e7f

expose studyset_studies

66f38b2

refactor

ef92015

fix cypress test

0e77732

suppress warnings with overlaps

e011869

try to fix ingestion test

01ea6aa

fix cypress test

277e332

use the stub_uuid

d39b0c9

fix more blind spots

9383f80

jdkent and others added 21 commits December 12, 2025 17:18

undo unnecessary changes

8869696

fix style issues

612f569

fix data check guards

8bd8c33

add script and fix tests

c16539c

fix style

8052574

make base study call more friendly

1949a9e

fix the frontend client api usage

96aa675

review of code

4278624

fix style

27f6369

fix style again

b9cea5a

remove redundant code

9f723e4

Apply suggestions from code review

c60150a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

fix line length issue

0b86b2c

do not use overlaps, be explicit about purpose of columns

33ff8c4

Update store/backend/migrations/versions/8e3f3d8a9b5b_add_curation_st…

d903f00

…ub_uuid.py

realign the basestudies endpoint

e284036

fix linting errors

82dfb37

fix: minor code improvements, remove unused code

34c8d7d

remove curation_stub_map from public api

7de1401

remove unused index

7841dc7

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

fix: remove unused imports

a4a063c

jdkent force-pushed the fix/sync_curation_extraction branch from e9a5cdc to a4a063c Compare December 12, 2025 23:19

jdkent added 4 commits December 12, 2025 18:53

fix backfill script

d17e769

assume postgres port

89cf2fe

arguments were not aligned to create a project

6423e8e

fix backfill script

055a464

jdkent merged commit c64151e into master Dec 13, 2025
18 checks passed

jdkent deleted the fix/sync_curation_extraction branch December 13, 2025 02:45

jdkent added this to Planning Dec 19, 2025

jdkent moved this to Done in Planning Dec 19, 2025

	(studyset) => API.NeurostoreServices.StudySetsService.studysetsPost(undefined, undefined, undefined, studyset),
	(studyset) => API.NeurostoreServices.StudySetsService.studysetsPost({ studyset }),

Conversation

jdkent commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nicoalee commented Dec 10, 2025

Uh oh!

nicoalee left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdkent commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

jdkent commented Dec 12, 2025

Uh oh!

nicoalee commented Dec 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jdkent commented Dec 6, 2025 •

edited

Loading

nicoalee left a comment •

edited

Loading

jdkent commented Dec 11, 2025 •

edited

Loading