Skip to content

Commit 6ba9e6a

Browse files
jaymedinajaymedina
authored
[SYNPY-1590] Implement documentation of Submission(+Status, +Bundle) OOP model (#1252)
* tdd initial tests * initial intro of dataclasses * expose api services for submission object * style and update docstring * add submission and submissionstatus models * add submission status retrieval and update methods; remove empty submissionstatus file * pipe query params directly into restAPI httpx requests * new dataclass object submission_bundle * move submission services functions to evaluation_services.py * renaming imports, to_synapse_request, request body refactor * patching up store method signature * update docs * new suite of tests * submissionstatus rework as a mutable object * bug fix for Statuses: updated to_synapse_request to follow same pattern as evaluations design * replace != with is not for full object comparison (not just keys) * expose the is_private arg for to_submission_status_annotations ONLY FOR submission annotations * fixed submission status/submission annotations * add support for legacy annotations * remove debug prints * get_all_submission_statuses now returns a list of substat objects * docstring updates * update submissionbundle docstrings, add more examples * initial sync test for status. moved evaluation_id retrieval to fill_from_dict for submissionstatus calls * update submissionBundle submissionstatus with evaluation_id * patch sync substatus integ tests. style. * fix submissionStatus integ tests and has_changed attribute * new substatus async integ tests. can_cancel can now be modified by an organizer on the client. cancel request returns no response body. * new test class for submission cancel functionality * substatus async unit tests * remove compare=false for some attributes. update sync unit tests * add submissionBundle integration tests * add submissionBundle unit tests * remove unnecessary imports and add style * get_evaluation_submissions returns generator object * get_user_submissions returns generator object * submissionBundle methods return generators * address final todo: implement docker_tag, and note in docs that version_number can be ignored for docker submissions * add async page in api references * add initial submission, status, bundle docs * updated tutorial purpose. add api reference to navbar * new tutorial scripts * remove try/excepts. add line references. add resources and source code sections. * add reference to File model * deprecate old submission model * style * add submission status and bundle to references * no need to import Submission for submission_organizer tutorial * status should be CLOSED not CANCELLED. add Submission import back (its for the delete step). * set tutorial global vars to None * fix line references * remove limits * add 2 more examples for Submission docstring * use the same evaluation_id in all examples * elaborate on submissions, statuses, bundles, and their relationship to evaluations * remove AcessControllable from being inherited
1 parent 60de6b0 commit 6ba9e6a

File tree

14 files changed

+681
-16
lines changed

14 files changed

+681
-16
lines changed

‎docs/reference/experimental/async/submission.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Submission
2+
3+
Contained within this file are experimental interfaces for working with the Synapse Python
4+
Client. Unless otherwise noted these interfaces are subject to change at any time. Use
5+
at your own risk.
6+
7+
## API Reference
8+
9+
::: synapseclient.models.Submission
10+
options:
11+
inherited_members: true
12+
members:
13+
- store_async
14+
- get_async
15+
- delete_async
16+
- cancel_async
17+
- get_evaluation_submissions_async
18+
- get_user_submissions_async
19+
- get_submission_count_async

‎docs/reference/experimental/async/submission_bundle.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Submission Bundle
2+
3+
Contained within this file are experimental interfaces for working with the Synapse Python
4+
Client. Unless otherwise noted these interfaces are subject to change at any time. Use
5+
at your own risk.
6+
7+
## API Reference
8+
9+
::: synapseclient.models.SubmissionBundle
10+
options:
11+
inherited_members: true
12+
members:
13+
- get_evaluation_submission_bundles_async
14+
- get_user_submission_bundles_async

‎docs/reference/experimental/async/submission_status.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Submission Status
2+
3+
Contained within this file are experimental interfaces for working with the Synapse Python
4+
Client. Unless otherwise noted these interfaces are subject to change at any time. Use
5+
at your own risk.
6+
7+
## API Reference
8+
9+
::: synapseclient.models.SubmissionStatus
10+
options:
11+
inherited_members: true
12+
members:
13+
- get_async
14+
- store_async
15+
- get_all_submission_statuses_async
16+
- batch_update_submission_statuses_async

‎docs/reference/experimental/sync/submission.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Submission
2+
3+
Contained within this file are experimental interfaces for working with the Synapse Python
4+
Client. Unless otherwise noted these interfaces are subject to change at any time. Use
5+
at your own risk.
6+
7+
## API Reference
8+
9+
::: synapseclient.models.Submission
10+
options:
11+
inherited_members: true
12+
members:
13+
- store
14+
- get
15+
- delete
16+
- cancel
17+
- get_evaluation_submissions
18+
- get_user_submissions
19+
- get_submission_count

‎docs/reference/experimental/sync/submission_bundle.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Submission Bundle
2+
3+
Contained within this file are experimental interfaces for working with the Synapse Python
4+
Client. Unless otherwise noted these interfaces are subject to change at any time. Use
5+
at your own risk.
6+
7+
## API Reference
8+
9+
::: synapseclient.models.SubmissionBundle
10+
options:
11+
inherited_members: true
12+
members:
13+
- get_evaluation_submission_bundles
14+
- get_user_submission_bundles

‎docs/reference/experimental/sync/submission_status.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Submission Status
2+
3+
Contained within this file are experimental interfaces for working with the Synapse Python
4+
Client. Unless otherwise noted these interfaces are subject to change at any time. Use
5+
at your own risk.
6+
7+
## API Reference
8+
9+
::: synapseclient.models.SubmissionStatus
10+
options:
11+
inherited_members: true
12+
members:
13+
- get
14+
- store
15+
- get_all_submission_statuses
16+
- batch_update_submission_statuses

‎docs/tutorials/python/submission.md‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Submissions, SubmissionStatuses, SubmissionBundles
2+
## What are Submissions?
3+
4+
In Synapse, a Submission is your entry to a challenge or evaluation queue. When you participate in a computational challenge or collaborative project, you submit your work (such as predictions, models, or analysis results) as a Submission to be evaluated and scored.
5+
6+
## Key Concepts
7+
8+
Before working with Submissions, it's helpful to understand how they fit into Synapse:
9+
10+
- **Entity**: Your actual work stored in Synapse (like a File containing predictions or a Docker image with your model)
11+
- **Evaluation**: A queue that accepts and organizes submissions for a specific challenge or project
12+
- **Submission**: The object associated with submitting your Entity to an Evaluation queue, creating a record that can be tracked and scored
13+
14+
## How Submissions Work
15+
16+
When you submit an Entity to an Evaluation:
17+
- The Submission creates an immutable record linking your Entity to that Evaluation
18+
- The Evaluation owns this Submission record (not you as the submitter)
19+
- Organizers can add scores and feedback through a SubmissionStatus object
20+
- You can track your Submissions and view their statuses
21+
22+
## Related Objects
23+
24+
The Python client provides three object types for working with Submissions:
25+
26+
- **Submission**: Represents your entry in an Evaluation queue
27+
- **SubmissionStatus**: Tracks scoring information and feedback for a Submission
28+
- **SubmissionBundle**: Combines a Submission and its SubmissionStatus for convenient access
29+
30+
## What You'll Learn
31+
32+
This tutorial covers two perspectives:
33+
34+
1. **Participating in challenges**: Making and tracking your submissions
35+
1. **Organizing challenges**: Scoring and managing submissions from participants
36+
37+
## Tutorial Purpose
38+
39+
In this tutorial:
40+
41+
As a participant of a Synapse challenge, you will
42+
43+
1. Make a submission to an existing evaluation queue on Synapse
44+
1. Fetch your existing submission
45+
1. Count your submissions
46+
1. Fetch all of your submissions from an existing evaluation queue on Synapse
47+
1. Check the status of your submission
48+
1. Cancel your submission
49+
50+
As an organizer of a Synapse challenge, you will
51+
52+
1. Annotate a submission to score it
53+
1. Batch-update submission statuses
54+
1. Fetch the submission bundle for a given submission
55+
1. Allow cancellation of submissions
56+
1. Delete submissions
57+
58+
## Prerequisites
59+
* You have completed the [Evaluation](./evaluation.md) tutorial, or have an existing Evaluation on Synapse to work from
60+
* You have an existing entity with which to make a submission (can be a [File](./file.md) or Docker Repository)
61+
* You have the correct permissions on the Evaluation queue for your desired tutorial section (participant or organizer)
62+
63+
## 1. Participating in a Synapse challenge
64+
65+
### 1. Make a submission to an existing evaluation queue on Synapse
66+
67+
```python
68+
{!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=32-54}
69+
```
70+
71+
### 2. Fetch your existing submission
72+
73+
```python
74+
{!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=56-71}
75+
```
76+
77+
### 3. Count your submissions
78+
79+
```python
80+
{!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=72-88}
81+
```
82+
83+
### 4. Fetch all of your submissions from an existing evaluation queue on Synapse
84+
85+
```python
86+
{!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=90-101}
87+
```
88+
89+
### 5. Check the status of your submission
90+
91+
```python
92+
{!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=103-125}
93+
```
94+
95+
### 6. Cancel your submission
96+
97+
```python
98+
{!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=126-143}
99+
```
100+
101+
## 2. Organizing a Synapse challenge
102+
103+
### 1. Annotate a submission to score it
104+
105+
```python
106+
{!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=33-60}
107+
```
108+
109+
### 2. Batch-update submission statuses
110+
111+
```python
112+
{!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=62-99}
113+
```
114+
115+
### 3. Fetch the submission bundle for a given submission
116+
117+
```python
118+
{!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=101-136}
119+
```
120+
121+
### 4. Allow cancellation of submissions
122+
123+
```python
124+
{!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=138-177}
125+
```
126+
127+
### 5. Delete submissions
128+
129+
```python
130+
{!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=179-209}
131+
```
132+
133+
## Source code for this tutorial
134+
135+
<details class="quote">
136+
<summary>Click to show me (source code for Participant)</summary>
137+
138+
```python
139+
{!docs/tutorials/python/tutorial_scripts/submission_participant.py!}
140+
```
141+
</details>
142+
143+
<details class="quote">
144+
<summary>Click to show me (source code for Organizer)</summary>
145+
146+
```python
147+
{!docs/tutorials/python/tutorial_scripts/submission_organizer.py!}
148+
```
149+
</details>
150+
151+
## References
152+
- [Evaluation][synapseclient.models.Evaluation]
153+
- [File][synapseclient.models.File]
154+
- [Submission][synapseclient.models.Submission]
155+
- [SubmissionStatus][synapseclient.models.SubmissionStatus]
156+
- [SubmissionBundle][synapseclient.models.SubmissionBundle]
157+
- [syn.login][synapseclient.Synapse.login]

0 commit comments

Comments
 (0)