-
Notifications
You must be signed in to change notification settings - Fork 73
[SYNPY-1590] Implement documentation of Submission(+Status, +Bundle) OOP model #1252
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
jaymedina
merged 57 commits into
synpy-1590-submission-model-main
from
synpy-1590-submission-model-docs
Dec 24, 2025
Merged
Changes from all commits
Commits
Show all changes
57 commits
Select commit
Hold shift + click to select a range
a061fb4
tdd initial tests
jaymedina d85a9d7
initial intro of dataclasses
jaymedina 50498b7
expose api services for submission object
jaymedina 94cedc6
style and update docstring
jaymedina cd19e0c
add submission and submissionstatus models
jaymedina 99cba20
add submission status retrieval and update methods; remove empty subm…
jaymedina 9ca2448
pipe query params directly into restAPI httpx requests
jaymedina f0d7ecc
new dataclass object submission_bundle
jaymedina 91b947b
move submission services functions to evaluation_services.py
jaymedina a6b46d9
renaming imports, to_synapse_request, request body refactor
jaymedina b68bd8e
patching up store method signature
jaymedina 5a44f07
update docs
jaymedina 65664d1
new suite of tests
jaymedina d214673
submissionstatus rework as a mutable object
jaymedina 2781797
bug fix for Statuses: updated to_synapse_request to follow same patte…
jaymedina 3230b1f
replace != with is not for full object comparison (not just keys)
jaymedina 2d43e5d
expose the is_private arg for to_submission_status_annotations ONLY F…
jaymedina d4d63ba
fixed submission status/submission annotations
jaymedina 8b9f9c9
add support for legacy annotations
jaymedina 6aa914a
remove debug prints
jaymedina 2d67fcc
get_all_submission_statuses now returns a list of substat objects
jaymedina c368e49
docstring updates
jaymedina fbd43f6
update submissionbundle docstrings, add more examples
jaymedina d5fac7e
initial sync test for status. moved evaluation_id retrieval to fill_f…
jaymedina 47797c0
update submissionBundle submissionstatus with evaluation_id
jaymedina e8e5361
patch sync substatus integ tests. style.
jaymedina 58ffe42
fix submissionStatus integ tests and has_changed attribute
jaymedina f34667d
new substatus async integ tests. can_cancel can now be modified by an…
jaymedina 5e762f0
new test class for submission cancel functionality
jaymedina 1bd7f4b
substatus async unit tests
jaymedina 4c8904d
remove compare=false for some attributes. update sync unit tests
jaymedina a223320
add submissionBundle integration tests
jaymedina f059751
add submissionBundle unit tests
jaymedina 3cb1915
remove unnecessary imports and add style
jaymedina 1fb0fbf
get_evaluation_submissions returns generator object
jaymedina 281ed3d
get_user_submissions returns generator object
jaymedina 1168cae
submissionBundle methods return generators
jaymedina a9977b2
address final todo: implement docker_tag, and note in docs that versi…
jaymedina 689fcb1
add async page in api references
jaymedina c54609b
add initial submission, status, bundle docs
jaymedina e0a79cc
updated tutorial purpose. add api reference to navbar
jaymedina 35efed7
new tutorial scripts
jaymedina e07206d
remove try/excepts. add line references. add resources and source cod…
jaymedina e193753
add reference to File model
jaymedina b710da0
deprecate old submission model
jaymedina 7917011
style
jaymedina 2021fb7
Merge branch 'synpy-1590-submission-model-main' into synpy-1590-submi…
jaymedina 4718119
add submission status and bundle to references
jaymedina 8510672
no need to import Submission for submission_organizer tutorial
jaymedina 727e302
status should be CLOSED not CANCELLED. add Submission import back (it…
jaymedina 15e450c
set tutorial global vars to None
jaymedina 7e57d72
fix line references
jaymedina 1982ba6
remove limits
jaymedina 2bd0304
add 2 more examples for Submission docstring
jaymedina 8cdb88c
use the same evaluation_id in all examples
jaymedina 985b727
elaborate on submissions, statuses, bundles, and their relationship t…
jaymedina fb6709c
remove AcessControllable from being inherited
jaymedina File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # Submission | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.Submission | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - store_async | ||
| - get_async | ||
| - delete_async | ||
| - cancel_async | ||
| - get_evaluation_submissions_async | ||
| - get_user_submissions_async | ||
| - get_submission_count_async | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Submission Bundle | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.SubmissionBundle | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - get_evaluation_submission_bundles_async | ||
| - get_user_submission_bundles_async |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Submission Status | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.SubmissionStatus | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - get_async | ||
| - store_async | ||
| - get_all_submission_statuses_async | ||
| - batch_update_submission_statuses_async |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # Submission | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.Submission | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - store | ||
| - get | ||
| - delete | ||
| - cancel | ||
| - get_evaluation_submissions | ||
| - get_user_submissions | ||
| - get_submission_count |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Submission Bundle | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.SubmissionBundle | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - get_evaluation_submission_bundles | ||
| - get_user_submission_bundles |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Submission Status | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.SubmissionStatus | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - get | ||
| - store | ||
| - get_all_submission_statuses | ||
| - batch_update_submission_statuses |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| # Submissions, SubmissionStatuses, SubmissionBundles | ||
jaymedina marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ## What are Submissions? | ||
|
|
||
| In Synapse, a Submission is your entry to a challenge or evaluation queue. When you participate in a computational challenge or collaborative project, you submit your work (such as predictions, models, or analysis results) as a Submission to be evaluated and scored. | ||
|
|
||
| ## Key Concepts | ||
|
|
||
| Before working with Submissions, it's helpful to understand how they fit into Synapse: | ||
|
|
||
| - **Entity**: Your actual work stored in Synapse (like a File containing predictions or a Docker image with your model) | ||
| - **Evaluation**: A queue that accepts and organizes submissions for a specific challenge or project | ||
| - **Submission**: The object associated with submitting your Entity to an Evaluation queue, creating a record that can be tracked and scored | ||
|
|
||
| ## How Submissions Work | ||
|
|
||
| When you submit an Entity to an Evaluation: | ||
| - The Submission creates an immutable record linking your Entity to that Evaluation | ||
| - The Evaluation owns this Submission record (not you as the submitter) | ||
| - Organizers can add scores and feedback through a SubmissionStatus object | ||
| - You can track your Submissions and view their statuses | ||
|
|
||
| ## Related Objects | ||
|
|
||
| The Python client provides three object types for working with Submissions: | ||
|
|
||
| - **Submission**: Represents your entry in an Evaluation queue | ||
| - **SubmissionStatus**: Tracks scoring information and feedback for a Submission | ||
| - **SubmissionBundle**: Combines a Submission and its SubmissionStatus for convenient access | ||
|
|
||
| ## What You'll Learn | ||
|
|
||
| This tutorial covers two perspectives: | ||
|
|
||
| 1. **Participating in challenges**: Making and tracking your submissions | ||
| 1. **Organizing challenges**: Scoring and managing submissions from participants | ||
|
|
||
| ## Tutorial Purpose | ||
|
|
||
| In this tutorial: | ||
|
|
||
| As a participant of a Synapse challenge, you will | ||
|
|
||
| 1. Make a submission to an existing evaluation queue on Synapse | ||
| 1. Fetch your existing submission | ||
| 1. Count your submissions | ||
| 1. Fetch all of your submissions from an existing evaluation queue on Synapse | ||
| 1. Check the status of your submission | ||
| 1. Cancel your submission | ||
|
|
||
| As an organizer of a Synapse challenge, you will | ||
|
|
||
| 1. Annotate a submission to score it | ||
| 1. Batch-update submission statuses | ||
| 1. Fetch the submission bundle for a given submission | ||
| 1. Allow cancellation of submissions | ||
| 1. Delete submissions | ||
|
|
||
| ## Prerequisites | ||
| * You have completed the [Evaluation](./evaluation.md) tutorial, or have an existing Evaluation on Synapse to work from | ||
| * You have an existing entity with which to make a submission (can be a [File](./file.md) or Docker Repository) | ||
| * You have the correct permissions on the Evaluation queue for your desired tutorial section (participant or organizer) | ||
|
|
||
| ## 1. Participating in a Synapse challenge | ||
|
|
||
| ### 1. Make a submission to an existing evaluation queue on Synapse | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=32-54} | ||
| ``` | ||
|
|
||
| ### 2. Fetch your existing submission | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=56-71} | ||
| ``` | ||
|
|
||
| ### 3. Count your submissions | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=72-88} | ||
| ``` | ||
|
|
||
| ### 4. Fetch all of your submissions from an existing evaluation queue on Synapse | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=90-101} | ||
| ``` | ||
|
|
||
| ### 5. Check the status of your submission | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=103-125} | ||
| ``` | ||
|
|
||
| ### 6. Cancel your submission | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=126-143} | ||
| ``` | ||
|
|
||
| ## 2. Organizing a Synapse challenge | ||
|
|
||
| ### 1. Annotate a submission to score it | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=33-60} | ||
| ``` | ||
|
|
||
| ### 2. Batch-update submission statuses | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=62-99} | ||
| ``` | ||
|
|
||
| ### 3. Fetch the submission bundle for a given submission | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=101-136} | ||
| ``` | ||
|
|
||
| ### 4. Allow cancellation of submissions | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=138-177} | ||
| ``` | ||
|
|
||
| ### 5. Delete submissions | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=179-209} | ||
| ``` | ||
|
|
||
| ## Source code for this tutorial | ||
|
|
||
| <details class="quote"> | ||
| <summary>Click to show me (source code for Participant)</summary> | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!} | ||
| ``` | ||
| </details> | ||
|
|
||
| <details class="quote"> | ||
| <summary>Click to show me (source code for Organizer)</summary> | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!} | ||
| ``` | ||
| </details> | ||
|
|
||
| ## References | ||
jaymedina marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - [Evaluation][synapseclient.models.Evaluation] | ||
| - [File][synapseclient.models.File] | ||
| - [Submission][synapseclient.models.Submission] | ||
| - [SubmissionStatus][synapseclient.models.SubmissionStatus] | ||
| - [SubmissionBundle][synapseclient.models.SubmissionBundle] | ||
| - [syn.login][synapseclient.Synapse.login] | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.