Skip to content

Commit f5f9bdf

Browse files
update ASReviewLAB tutorial (#28)
Co-authored-by: Rens van de schoot <36502709+Rensvandeschoot@users.noreply.github.com>
1 parent f6d20fb commit f5f9bdf

31 files changed

+168
-208
lines changed

ASReviewLAB.qmd

Lines changed: 53 additions & 92 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,12 @@ title: "Introductory exercise to ASReview LAB"
33
author: "The ASReview Academy Team"
44
---
55

6-
## Introduction to the software ASReview LAB
6+
**This exercise was made using ASReview LAB version 1.6. If you have a different version you can observe differences between your setup and the exercise.**
7+
8+
## Introduction to the software ASReview LAB v1.6
79

810
The **goal** of this exercise is to get familiar with AI-aided screening
9-
by making use of ASReview LAB v1.x.
11+
by making use of ASReview LAB v1.6.
1012

1113
You will learn how to install and set up the software, upload data,
1214
screen records, and export and interpret the results. The exercise will
@@ -29,92 +31,50 @@ Before you start, you might want to read a bit more on:
2931
- The paper that was published in [Nature Machine
3032
Intelligence](https://www.nature.com/articles/s42256-020-00287-7){target="_blank"}
3133

32-
## The software
33-
34-
### Step 1: Installing Python and ASReview LAB
35-
36-
First, you need to install [Python](https://www.python.org/downloads/){target="_blank"}.
37-
38-
Once you have Python installed, you can go through the easy 3-step
39-
[guide to installing (or upgrading)
40-
ASReview](https://asreview.nl/download/){target="_blank"} on the ASReview-website.
41-
42-
More detailed installation information, troubleshooting, and
43-
installation on a server or via a Docker are available on
44-
[ReadTheDocs](https://asreview.readthedocs.io/en/latest/installation.html#install-asreview){target="_blank"}.
45-
46-
*Have you installed the latest version of ASReview? You can proceed to
47-
step 2!*
48-
49-
### Step 2: Starting ASReview LAB
50-
51-
To open ASReview LAB in your browser, you need to start it in the
52-
command prompt (i.e. CMD.exe or Terminal). You can open your command
53-
prompt by typing ‘cmd’ (for Windows) or ‘terminal’ (for Apple) in your
54-
computer’s search bar (select ‘Run as administrator’ if you have this
55-
option).
56-
57-
![](images/ASReviewLAB/CMD1.png){width=95%}
58-
59-
The command prompt will open, in which you can type the following
60-
command and press enter:
34+
## Prerequisites
6135

62-
``` bash
63-
asreview lab
64-
```
36+
### Installing ASReview LAB
6537

66-
![](images/ASReviewLAB/CMD2.png){width=95%}
38+
Before you start the tutorial ASReview LAB needs to be installed; see for instructions the [ASReview website](https://asreview.nl/download/){target="_blank"}.
6739

68-
It takes a few seconds for ELAS - your Electronic Learning Assistant -
69-
to start the software. It will appear in your (default) web browser.
40+
*Have you installed ASReview LAB? You can proceed to the exercise!*
7041

71-
But why do you need to start it up by running code in your command
72-
prompt? This ensures that ASReview LAB is run locally. More
73-
specifically, <u>your data is and stays your own</u>. Small price to pay
74-
for complete privacy, right?! Read more about the key principles in the
75-
[Zen of Elas](https://asreview.nl/blog/the-zen-of-elas/){target="_blank"}!
42+
## Exercise
7643

77-
Note that you have to keep your command-line interpreter running while
78-
using ASReview LAB, even though the interface is in your browser!
79-
80-
You can also run the software via a server, but you need to take care of
81-
[hosting the server
82-
yourself](https://asreview.readthedocs.io/en/latest/installation.html#server-installation){target="_blank"}
83-
(or ask your IT-department).
44+
### Step 1: Starting ASReview LAB
8445

46+
Open ASReview LAB in your browser. Note that if you do this via Command Prompt (Windows) or Terminal (MacOS) you have to keep your command-line interpreter running while using ASReview LAB, even though the interface is in your browser!
8547

8648
*Have you opened ASReview LAB in your browser? If so, you can proceed to
87-
step 3!*
49+
step 2!*
8850

89-
### Step 3: Creating a project
51+
### Step 2: Creating a project
9052

91-
Now that you have installed and opened ASReview LAB, you can create a
92-
new project. Below you will find a step-by-step guide. Note that the
93-
screenshots shown below are made in [dark
94-
mode](https://asreview.readthedocs.io/en/latest/screening.html#display){target="_blank"}.
53+
Now that you have opened ASReview LAB, you can create a
54+
new project. Below you will find a step-by-step guide.
9555

9656
1. *New project;*
9757

98-
Hover your mouse over the ‘create’ button with the plus sign in the
58+
Hover your mouse over the ‘`create`’ button with the plus sign in the
9959
bottom right corner.
10060

101-
![](images/ASReviewLAB/Interface2.png){width=95%}
61+
![](images/ASReviewLAB/step_2a_V1_6.png){width=85% fig-align="center"}
10262

10363
2. *Project name;*
10464

105-
Select Validation Mode, fill out a project name and press ‘NEXT’. Note
65+
Select Validation Mode, fill out a project name and press ‘`NEXT`’. Note
10666
that you can fill out your name and a description as well.
10767

108-
![](https://raw.githubusercontent.com/asreview/asreview/main/docs/images/setup_project_modes.png){width=95%}
68+
![](images/ASReviewLAB/step_2b_V1_6.png){width=85% fig-align="center"}
10969

110-
For this exercise we are screening in the so-called ‘Validation Mode’
70+
For this exercise we are screening in the so-called ‘`Validation Mode`
11171
of ASReview. By screening in the [Validation
11272
Mode](https://asreview.readthedocs.io/en/latest/project_create.html#project-modes){target="_blank"},
11373
we are going to make use of a [benchmark
11474
dataset](https://asreview.readthedocs.io/en/latest/data_labeled.html#fully-labeled-data){target="_blank"}.
11575
This means that all records in the dataset have already been labeled as
11676
relevant or irrelevant. This is indicated to the user through a banner
117-
above each article. Note that in ‘Oracle Mode’ - when screening your own
77+
above each article. Note that in ‘`Oracle Mode`’ - when screening your own
11878
dataset - the relevant papers will not be marked; you, the oracle, have
11979
to make the decisions.
12080

@@ -123,33 +83,33 @@ More detailed information about setting up a project can be found on
12383

12484

12585
*Have you started creating a new project? If so, you can proceed to step
126-
4!*
86+
3!*
12787

12888
## Project setup
12989

130-
### Step 4: The dataset
90+
### Step 3: The dataset
13191

13292
Now that you have created your ASReview project (woohoo!), you need to set it
13393
up. Without data, we have nothing to screen. So, you need to tell ELAS
13494
which dataset you want to screen for relevant articles.
13595

136-
Click on the ‘ADD’ button next to ‘Add dataset’. Now a menu appears
96+
Click on the ‘`ADD`’ button next to ‘`Add dataset`’. Now a menu appears
13797
where you can choose how to load the dataset. You can add your dataset
13898
by selecting a file or providing an URL. For this exercise, we will use
13999
a benchmark dataset.
140100

141-
Go to the ‘Benchmark datasets’ button, open the first dataset (i.e. the
101+
Go to the ‘`Benchmark datasets`’ button, open the first dataset (i.e. the
142102
Van de Schoot (2017) dataset about PTSD trajectories) and click on
143-
SELECT’. After you select the dataset, click on ‘SAVE’.
103+
`ADD`’. After you select the dataset, click on ‘`SAVE`’.
144104

145-
![](images/ASReviewLAB/Interface5.png){width=95%}
105+
![](images/ASReviewLAB/step_3_V1_6.png){width=85% fig-align="center"}
146106

147107

148108

149109
*Have you successfully selected/uploaded the dataset? If so, you can
150-
proceed to step 5!*
110+
proceed to step 4!*
151111

152-
### Step 5: Prior knowledge
112+
### Step 4: Prior knowledge
153113

154114
Before you can start screening the records, you need to tell ELAS what
155115
kind of records you <u>are</u> and what kind of records you <u>are
@@ -165,16 +125,15 @@ irrelevant record). However, because you are using the Validation Mode
165125
of ASReview, the relevant records are known; the original authors have
166126
already read ALL records.
167127

168-
To select the prior knowledge you first need to click on the ‘ADD’
169-
button next to ‘Add prior knowledge’; see also the documentation
128+
To select the prior knowledge you first need to click on the ‘`ADD`
129+
button next to ‘`Add prior knowledge`’; see also the documentation
170130
about the selection of [prior knowledge](https://asreview.readthedocs.io/en/latest/project_create.html#select-prior-knowledge){target="_blank"}.
171131
Now you will see a menu about selecting prior knowledge.
172132

173133
The following five papers are known to be relevant:
174134

175-
- Latent trajectories of trauma symptoms and resilience: the 3-year
176-
longitudinal prospective USPER study of Danish veterans deployed in
177-
Afghanistan (DOI: [10.4088/JCP.13m08914](https://doi.org/10.4088/jcp.13m08914){target="_blank"})
135+
- Latent Trajectories of Trauma Symptoms and Resilience
136+
(DOI: [10.4088/JCP.13m08914](https://doi.org/10.4088/jcp.13m08914){target="_blank"})
178137
- A Latent Growth Mixture Modeling Approach to PTSD Symptoms in Rape
179138
Victims (DOI: [10.1177/1534765610395627](https://doi.org/10.1177/1534765610395627){target="_blank"})
180139
- Peace and War: Trajectories of Posttraumatic Stress Disorder
@@ -185,22 +144,22 @@ The following five papers are known to be relevant:
185144
- Trajectories of trauma symptoms and resilience in deployed US
186145
military service members: Prospective cohort study (DOI: [10.1192/bjp.bp.111.096552](https://doi.org/10.1192/bjp.bp.111.096552){target="_blank"})
187146

188-
To add the relevant records, you click on ‘Search’, copy and paste the titles
147+
To add the relevant records, you click on ‘`Search`’, copy and paste the titles
189148
of these relevant records one by one in the search bar and add them as
190149
relevant.
191150

192-
![](https://raw.githubusercontent.com/asreview/asreview/main/docs/images/setup_prior_knowledge_random_validate.png){width=95%}
151+
![](images/ASReviewLAB/step_4_V1_6.png){width=85% fig-align="center"}
193152

194153

195154
After adding all five relevant records, you can add some irrelevant ones
196-
by clicking the ‘Random’ button (use the arrow in the upper left corner
197-
to be able to select this button) and by changing ‘relevant’ to
198-
‘irrelevant’. Select five irrelevant records and click on ‘CLOSE’.
155+
by clicking the ‘`Random`’ button (use the arrow in the upper left corner
156+
to be able to select this button) and by changing ‘`relevant`’ to
157+
`irrelevant`’. Select five irrelevant records and click on ‘`CLOSE`’.
199158

200159
*Have you selected five relevant and five irrelevant records? If so, you
201-
can proceed to step 6!*
160+
can proceed to step 5!*
202161

203-
### Step 6: Active learning model
162+
### Step 5: Active learning model
204163

205164
The last step to complete the setup is to select the active learning
206165
model you want to use. The default settings (i.e. Naïve Bayes, Max and
@@ -210,8 +169,8 @@ read which options are
210169
or add your own model via a
211170
[template](https://github.com/asreview/template-extension-new-model){target="_blank"}.
212171

213-
You can click on ‘NEXT’. A menu with the defaults will appear. Since we
214-
are using the defaults, you can click on ‘NEXT’ again. In the last step
172+
You can click on ‘`NEXT`’. A menu with the defaults will appear. Since we
173+
are using the defaults, you can click on ‘`NEXT`’ again. In the last step
215174
of the setup, ASReview LAB runs the feature extractor, trains a model,
216175
and ranks the records in your dataset. Depending on the model and the
217176
size of your dataset, this can take a couple of minutes (meanwhile, you
@@ -220,13 +179,15 @@ Learning?](https://asreview.nl/blog/active-learning-explained/){target="_blank"}
220179

221180
After the project is successfully initialized, you can start reviewing.
222181

223-
![](images/ASReviewLAB/Interface9.png){width=95%}
182+
![](images/ASReviewLAB/step_5a_V1_6.png){width=85% fig-align="center"}
183+
184+
*Have you finished the setup? If so, you can proceed to step 6!*
224185

225-
*Have you finished the setup? If so, you can proceed to step 7!*
186+
![](images/ASReviewLAB/step_5b_V1_6.png){width=85% fig-align="center"}
226187

227188
## Screening phase
228189

229-
### Step 7: Screening the records
190+
### Step 6: Screening the records
230191

231192
Everything is set up and ready to screen, well done!
232193

@@ -254,7 +215,7 @@ statistics and interpret the corresponding charts, check out the
254215
[documentation](https://asreview.readthedocs.io/en/latest/progress.html#analytics){target="_blank"}
255216
on the Analytics page.
256217

257-
The Van de Schoot (2017) dataset contained 43 relevant records in this
218+
The Van de Schoot (2017) dataset contained 38 relevant records in this
258219
particular example. Did you get to label all of them as relevant before
259220
you reached your Stopping Rule? If you did, great!
260221

@@ -278,14 +239,14 @@ Rule. Read more about Stopping Rules and how to decide on a good
278239
strategy for your data on the [discussion
279240
platform](https://github.com/asreview/asreview/discussions){target="_blank"}.
280241

281-
### Step 8: Extracting and inspecting the data
242+
### Step 7: Extracting and inspecting the data
282243

283244
Now that you found all or most relevant records, you can export your
284245
data using [these
285246
instructions](https://asreview.readthedocs.io/en/latest/progress.html#export-results){target="_blank"}.
286247
If you choose to inspect your data in Excel, download the data in
287-
‘Excel’ format. If you prefer to inspect your data in R, download the
288-
‘CSV (UTF-8)’ format and open it in R.
248+
`Excel`’ format. If you prefer to inspect your data in R, download the
249+
`CSV (UTF-8)`’ format and open it in R.
289250

290251
You can find all the data that was originally imported to ASReview in
291252
the exported data file, in a new order and with two new columns added at
@@ -302,7 +263,7 @@ the original ordering, do the included articles come from?
302263

303264
For the last exercise, it is important to change the order back to the order
304265
provided by ASReview. Lastly, check out the first few records with no number
305-
in the `included` column. Are those articles labeled as ‘relevant’ in the
266+
in the `included` column. Are those articles labeled as ‘`relevant`’ in the
306267
original dataset? (Whether or not a record was pre-labeled as relevant is
307268
shown in the column `label_included` in the original dataset.)
308269

@@ -342,8 +303,8 @@ Some suggestions:
342303
code](https://github.com/asreview/asreview/tree/master/asreview){target="_blank"} on
343304
Github… +1 for open-science!)
344305

306+
![](images/ASReviewLAB/game_V1_6.png){width=85% fig-align="center"}
345307

346-
![](https://raw.githubusercontent.com/asreview/asreview/main/docs/images/game.png){width=95%}
347308

348309
[^1]: Wang Z, Nayfeh T, Tetzlaff J, O’Blenis P, Murad MH (2020) Error
349310
rates of human reviewers during abstract screening in systematic

_quarto.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ website:
2626
- section: "Introduction Courses"
2727
contents:
2828
- href: ASReviewLAB.qmd
29-
text: ASReview LAB
29+
text: ASReview LAB v1.6
3030
- href: datatools.qmd
3131
text: Datatools
3232
- href: simulation.qmd

0 commit comments

Comments
 (0)