docs: DOC-282: Pause an annotator (#7147)

Co-authored-by: caitlinwheeless <[email protected]>

Author: caitlinwheeless

docs/source/guide/quality.md

@@ -116,6 +116,34 @@ For specific labels, you can see in a donut chart how many labels of each type w
 
 For example, if you're developing a dataset of OCR images, and 90% of your tasks have **Text** labels and 10% have **Handwriting** labels, you might want to increase the number of images of handwriting in your dataset, to improve the eventual accuracy of a machine learning model trained on this dataset.
 
+## Pause an annotator
+
+For organizations with a large number of annotators, it might prove useful to pause an annotator's progress. This might be useful for annotators that are performing poorly or exhibiting behavior that might indicate they have automated their work (bot behavior). 
+
+You can pause annotators from the Members dashboard in a project. This action is only available next to users in the Annotator and Reviewer roles:
+
+![Screenshot of pause](/images/review/pause.png)
+
+When a user is paused, the following occurs:
+
+* They immediately see a message informing them that they have been paused. 
+
+    ![Screenshot of message](/images/review/paused-message.png)
+* Their progress within their current task is saved as a draft, but they cannot make any further changes.   
+* When they click **Go Back**, they are returned to the Projects page. If they attempt to re-enter the project, they are shown the error message above. 
+
+!!! info Tip
+
+    If you have [custom scripts](scripts) enabled, you can automatically pause an annotator based on certain behaviors and then customize the message that appears on their screen. 
+
+    For more information, see [Custom script examples - Pause an annotator](script_examples#Pause-an-annotator).
+
+!!! info Tip
+
+    If you hover over the **Paused** indicator, you can see the message that was shown to the user when they were paused. If a user was manually paused, it also shows who initiated the action.  
+
+    ![Screenshot of hover](/images/review/paused-tooltip.png)
+
 ## Verify model and annotator performance
 
 To verify the performance of specific annotators, review the **Members** section for a specific project. If you don't see an annotator's activity reflected, make sure they have been added as a member to the project.

docs/source/guide/script_examples.md

@@ -15,6 +15,9 @@ The following examples work when [custom scripts](scripts) are enabled.
 
 For details on implementing your own custom scripts, see [Label Studio Interface (LSI)](scripts#Label-Studio-Interface-LSI) and [Frontend API implementation details](scripts#Frontend-API-implementation-details). 
 
+!!! info Tip
+    You can find additional script examples in our [label-studio-custom-scripts repo](https://github.com/HumanSignal/label-studio-custom-scripts).
+
 ## Plotly
 
 Use [Plotly](https://plotly.com/) to insert charts and graphs into your labeling interface. Charts are rendered in every annotation opened by a user. 
@@ -597,4 +600,223 @@ The `TimelineLabels` tag is connected to the second video (`video0`), allowing a
     "video_url": "https://example.com/path/to/video.mp4"
   }
 }
-```
+```
+
+## Pause an annotator
+
+You can manually [pause an annotator](quality#Pause-an-annotator) to prevent stop them from completing tasks and revoke their project access. 
+
+This script automatically pauses an annotator who breaks any of the following rules and customizes the message that appears:
+
+* Too many duplicate values `timesInARow(3)`:
+
+    Checks if the last three submitted annotations in the `TextArea` field (`comment`) all have the same value. If they do, it returns a custom warning message. 
+
+    ![Screenshot of warning](/images/project/scripts_pause1.png)
+
+* Too many similar values `tooSimilar()`: 
+
+    For the `Choices` options (`sentiment`), it computes a deviation over the past values. If the deviation is below a threshold (meaning the values are too uniform/similar), it returns a custom warning message. 
+
+    ![Screenshot of warning](/images/project/scripts_pause2.png)
+ 
+* Too many submissions over a period of time `tooFast()`: 
+
+    Monitors the overall speed of annotations. It checks if, for example, 20 annotations were submitted in less than 10 minutes. If so, a custom warning appears. 
+
+    ![Screenshot of warning](/images/project/scripts_pause3.png)
+
+To unpause an annotator, use the [Members dashboard](quality#Pause-an-annotator). 
+
+!!! info Tip
+
+    If you hover over the **Paused** indicator, you can see the message that was shown to the user when they were paused. If a user was manually paused, it also shows who initiated the action.  
+
+    ![Screenshot of hover](/images/project/scripts_pause_hover.png)
+
+#### Script
+
+```javascript
+/****** CONFIGURATION FOR PAUSING RULES ******/
+/**
+ * `fields` describe per-field rules in a format
+ *   <field-name>: [<rule>(<optional params for the rule>)]
+ * `global` is for rules applied to the whole annotation
+ */
+const RULES = {
+  fields: {
+    comment: [timesInARow(3)],
+    sentiment: [tooSimilar()],
+  },
+  global: [tooFast()],
+}
+/**
+ * Messages for users when they are paused.
+ * Each message is a function with the same name as original rule and it receives an object with
+ * `items` and `field`.
+ */
+const MESSAGES = {
+  timesInARow: ({ field }) => `Too many similar values for ${field}`,
+  tooSimilar: ({ field }) => `Too similar values for ${field}`,
+  tooFast: () => `Too fast annotations`,
+}
+
+
+
+/****** ALL AVAILABLE RULES ******/
+/**
+ * They recieve params and return function which recieves `items` and optional `field`.
+ * If condition is met it returns warning message. If not — returns `false`.
+ */
+
+// check if values for the `field` in last `times` items are the same
+function timesInARow(times) {
+  return (items, field) => {
+    if (items.length < times) return false
+    const last = String(items.at(-1).values[field])
+    return items.slice(-times).every((item) => String(item.values[field]) === last)
+      ? MESSAGES.timesInARow({ items, field })
+      : false
+  };
+}
+function tooSimilar(deviation = 0.1, max_count = 10) {
+  return (items, field) => {
+    if (items.length < max_count) return false
+    const values = items.map((item) => item.values[field])
+    const points = values.map((v) => values.indexOf(v))
+    return calcDeviation(points) < deviation
+      ? MESSAGES.tooSimilar({ items, field })
+      : false
+  };
+}
+function tooFast(minutes = 10, times = 20) {
+  return (items) => {
+    if (items.length < times) return false
+    const last = items.at(-1)
+    const first = items.at(-times)
+    return last.created_at - first.created_at < minutes * 60
+      ? MESSAGES.tooFast({ items })
+      : false
+  };
+}
+
+/****** INTERNAL CODE ******/
+const project = DM.project.id
+if (!DM.project) return;
+
+const key = ["__pause_stats", project].join("|")
+const fields = Object.keys(RULES.fields)
+// { sentiment: ["positive", ...], comment: undefined }
+const values = Object.fromEntries(fields.map(
+  (field) => [field, DM.project.parsed_label_config[field]?.labels],
+))
+
+// simplified version of MSE with normalized x-axis
+function calcDeviation(data) {
+  const n = data.length;
+  // we normalize indices from -n/2 to n/2 so meanX is 0
+  const mid = n / 2;
+  const mean = data.reduce((a, b) => a + b) / n;
+
+  const k = data.reduce((a, b, i) => a + (b - mean) * (i - mid), 0) / data.reduce((a, b, i) => a + (i - mid) ** 2, 0);
+  const mse = data.reduce((a, b, i) => a + (b - (k * (i - mid) + mean)) ** 2, 0) / n;
+
+  return Math.abs(mse);
+}
+
+LSI.on("submitAnnotation", (_store, ann) => {
+  const results = ann.serializeAnnotation()
+  // { sentiment: "positive", comment: "good" }
+  const values = {}
+  fields.forEach((field) => {
+    const value = results.find((r) => r.from_name === field)?.value
+    if (!value) return;
+    if (value.choices) values[field] = value.choices.join("|")
+    else if (value.text) values[field] = value.text
+  })
+  let stats = []
+  try {
+    stats = JSON.parse(localStorage.getItem(key)) ?? []
+  } catch(e) {}
+  stats.push({ values, created_at: Date.now() / 1000 })
+
+  for (const rule of RULES.global) {
+    const result = rule(stats)
+    if (result) {
+      localStorage.setItem(key, "[]");
+      pause(result);
+      return;
+    }
+  }
+
+  for (const field of fields) {
+    if (!values[field]) continue;
+    for (const rule of RULES.fields[field]) {
+      const result = rule(stats, field)
+      if (result) {
+        localStorage.setItem(key, "[]");
+        pause(result);
+        return;
+      }
+    }
+  }
+
+  localStorage.setItem(key, JSON.stringify(stats));
+});
+
+function pause(verbose_reason) {
+  const body = {
+    reason: "CUSTOM_SCRIPT",
+    verbose_reason,
+  }
+  const options = {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify(body),
+  }
+  fetch(`/api/projects/${project}/members/${Htx.user.id}/pauses`, options)
+}
+```
+
+**Related LSI instance methods:**
+
+* [on(eventName, handler)](scripts#on-eventName-handler)
+  
+**Related frontend events:**
+
+* [submitAnnotation](frontend_reference#submitAnnotationn)
+
+#### Labeling config
+
+This labeling config presents users with text and asks them to:
+
+* Provide a sentiment value using `<Choices>`
+* Comment on their reasoning using `<TextArea>`
+
+```xml
+<View>
+  <Text name="text" value="$text"/>
+  <View style="box-shadow: 2px 2px 5px #999; padding: 20px; margin-top: 2em; border-radius: 5px;">
+    
+    <Header value="What is the sentiment of this text?" />
+    <Choices name="sentiment" toName="text" choice="single" showInLine="true">
+      <Choice value="positive" hotkey="1" />
+      <Choice value="negative" hotkey="2" />
+      <Choice value="neutral" hotkey="3" />
+    </Choices>
+
+    <Header value="Why?" />
+    <TextArea name="comment" toName="text" rows="4" placeholder="Add your comment here..." />
+  
+  </View>
+</View>
+
+```
+
+**Related tags:**
+
+* [View](/tags/view.html)
+* [Text](/tags/text.html)
+* [Header](/tags/header.html)
+* [Choices](/tags/choices.html)
+* [TextArea](/tags/textarea.html)