Skip to content

Commit 316a254

Browse files
caitlinwheelesscaitlinwheeless
and
caitlinwheeless
authored
docs: DOC-285: Annotation Limit settings (#7236)
Co-authored-by: caitlinwheeless <[email protected]>
1 parent 338fe8e commit 316a254

File tree

3 files changed

+44
-7
lines changed

3 files changed

+44
-7
lines changed

docs/source/guide/project_settings_lse.md

+28
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,34 @@ The following options supersede what you specified under [**Annotations > Task S
454454

455455
</dd>
456456

457+
<dt id="annotation-limit">Annotation Limit</dt>
458+
459+
<dd>
460+
461+
Set limits on how many tasks each individual user can annotate. This can be useful if you are concerned with preventing any potential bias that might arise from a small set of power users completing a majority of project tasks.
462+
463+
When an annotator reaches their limit, they will see a notification telling them that they have been paused. When paused, an annotator can no longer access the project.
464+
465+
| Field | Description |
466+
| ------------- | ------------ |
467+
| **Limit by Number of Tasks** | Set a specific number of tasks each annotator is able to complete before their progress is paused. |
468+
| **Limit by Percentage of Tasks** | Calculate the number of tasks each annotator is able to complete as a percentage of tasks within the project. |
469+
470+
You can set one or both values. Annotators will be paused when they reach whichever limit is smaller.
471+
472+
To unpause annotators:
473+
474+
* You can manually unpause annotators using the **Paused** toggle available in the Members dashboard.
475+
* If you increase the limits, previously paused annotators will regain access.
476+
* If you are using a percentage-based limit and you add more tasks to a project, previously paused annotators will regain access.
477+
478+
For more information about pausing annotators, including how to manually pause specific annotators, see [Pause an annotator](quality#Pause-an-annotator).
479+
480+
!!! note
481+
Pauses affect users in Annotator and Reviewer roles. So, for example, if a Reviewer is also annotating tasks and they hit the annotation limit, they will be unable to regain access to the project to review annotations unless they are unpaused.
482+
483+
</dd>
484+
457485
<dt id="task-agreement">Task Agreement</dt>
458486

459487
<dd>

docs/source/guide/quality.md

+16-7
Original file line numberDiff line numberDiff line change
@@ -118,9 +118,11 @@ For example, if you're developing a dataset of OCR images, and 90% of your tasks
118118

119119
## Pause an annotator
120120

121-
For organizations with a large number of annotators, it might prove useful to pause an annotator's progress. This might be useful for annotators that are performing poorly or exhibiting behavior that might indicate they have automated their work (bot behavior).
121+
For organizations with a large number of annotators, it might prove useful to pause an annotator's progress. This might be helpful for annotators that are performing poorly or exhibiting behavior that might indicate they have automated their work (bot behavior).
122122

123-
You can pause annotators from the Members dashboard in a project. This action is only available next to users in the Annotator and Reviewer roles:
123+
### Manually pause an annotator
124+
125+
You can manually pause annotators from the Members dashboard in a project. This action is only available next to users in the Annotator and Reviewer roles:
124126

125127
![Screenshot of pause](/images/review/pause.png)
126128

@@ -134,15 +136,22 @@ When a user is paused, the following occurs:
134136

135137
!!! info Tip
136138

137-
If you have [custom scripts](scripts) enabled, you can automatically pause an annotator based on certain behaviors and then customize the message that appears on their screen.
139+
If you hover over the **Paused** indicator, you can see the message that was shown to the user when they were paused. If a user was manually paused, it also shows who initiated the action.
138140

139-
For more information, see [Custom script examples - Pause an annotator](script_examples#Pause-an-annotator).
141+
![Screenshot of hover](/images/review/paused-tooltip.png)
140142

141-
!!! info Tip
143+
### Automatically pause annotators
142144

143-
If you hover over the **Paused** indicator, you can see the message that was shown to the user when they were paused. If a user was manually paused, it also shows who initiated the action.
145+
#### Annotation Limit settings
146+
147+
You can use **Settings > Quality > Annotation Limit** to set limits on how many tasks an annotator is able to complete before they are paused. For more information, see [Annotation Limit](project_settings_lse#annotation-limit).
148+
149+
#### Behavior-based triggers
150+
151+
If you have [custom scripts](scripts) enabled, you can automatically pause an annotator based on certain behaviors and then customize the message that appears on their screen.
152+
153+
For more information, see [Custom script examples - Pause an annotator](script_examples#Pause-an-annotator).
144154

145-
![Screenshot of hover](/images/review/paused-tooltip.png)
146155

147156
## Verify model and annotator performance
148157

Loading

0 commit comments

Comments
 (0)