You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: docs/source/guide/project_settings_lse.md
+28
Original file line number
Diff line number
Diff line change
@@ -454,6 +454,34 @@ The following options supersede what you specified under [**Annotations > Task S
454
454
455
455
</dd>
456
456
457
+
<dtid="annotation-limit">Annotation Limit</dt>
458
+
459
+
<dd>
460
+
461
+
Set limits on how many tasks each individual user can annotate. This can be useful if you are concerned with preventing any potential bias that might arise from a small set of power users completing a majority of project tasks.
462
+
463
+
When an annotator reaches their limit, they will see a notification telling them that they have been paused. When paused, an annotator can no longer access the project.
464
+
465
+
| Field | Description |
466
+
| ------------- | ------------ |
467
+
|**Limit by Number of Tasks**| Set a specific number of tasks each annotator is able to complete before their progress is paused. |
468
+
|**Limit by Percentage of Tasks**| Calculate the number of tasks each annotator is able to complete as a percentage of tasks within the project. |
469
+
470
+
You can set one or both values. Annotators will be paused when they reach whichever limit is smaller.
471
+
472
+
To unpause annotators:
473
+
474
+
* You can manually unpause annotators using the **Paused** toggle available in the Members dashboard.
475
+
* If you increase the limits, previously paused annotators will regain access.
476
+
* If you are using a percentage-based limit and you add more tasks to a project, previously paused annotators will regain access.
477
+
478
+
For more information about pausing annotators, including how to manually pause specific annotators, see [Pause an annotator](quality#Pause-an-annotator).
479
+
480
+
!!! note
481
+
Pauses affect users in Annotator and Reviewer roles. So, for example, if a Reviewer is also annotating tasks and they hit the annotation limit, they will be unable to regain access to the project to review annotations unless they are unpaused.
Copy file name to clipboardexpand all lines: docs/source/guide/quality.md
+16-7
Original file line number
Diff line number
Diff line change
@@ -118,9 +118,11 @@ For example, if you're developing a dataset of OCR images, and 90% of your tasks
118
118
119
119
## Pause an annotator
120
120
121
-
For organizations with a large number of annotators, it might prove useful to pause an annotator's progress. This might be useful for annotators that are performing poorly or exhibiting behavior that might indicate they have automated their work (bot behavior).
121
+
For organizations with a large number of annotators, it might prove useful to pause an annotator's progress. This might be helpful for annotators that are performing poorly or exhibiting behavior that might indicate they have automated their work (bot behavior).
122
122
123
-
You can pause annotators from the Members dashboard in a project. This action is only available next to users in the Annotator and Reviewer roles:
123
+
### Manually pause an annotator
124
+
125
+
You can manually pause annotators from the Members dashboard in a project. This action is only available next to users in the Annotator and Reviewer roles:
124
126
125
127

126
128
@@ -134,15 +136,22 @@ When a user is paused, the following occurs:
134
136
135
137
!!! info Tip
136
138
137
-
If you have [custom scripts](scripts) enabled, you can automatically pause an annotator based on certain behaviors and then customize the message that appears on their screen.
139
+
If you hover over the **Paused** indicator, you can see the message that was shown to the user when they were paused. If a user was manually paused, it also shows who initiated the action.
138
140
139
-
For more information, see [Custom script examples - Pause an annotator](script_examples#Pause-an-annotator).
141
+

140
142
141
-
!!! info Tip
143
+
### Automatically pause annotators
142
144
143
-
If you hover over the **Paused** indicator, you can see the message that was shown to the user when they were paused. If a user was manually paused, it also shows who initiated the action.
145
+
#### Annotation Limit settings
146
+
147
+
You can use **Settings > Quality > Annotation Limit** to set limits on how many tasks an annotator is able to complete before they are paused. For more information, see [Annotation Limit](project_settings_lse#annotation-limit).
148
+
149
+
#### Behavior-based triggers
150
+
151
+
If you have [custom scripts](scripts) enabled, you can automatically pause an annotator based on certain behaviors and then customize the message that appears on their screen.
152
+
153
+
For more information, see [Custom script examples - Pause an annotator](script_examples#Pause-an-annotator).
144
154
145
-

0 commit comments