You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# this has an effect in exporting when from is set to 'source'
79
+
# and also in metrics calculations where service is considered
80
+
published: true
69
81
82
+
# Use the EOSC-Marketplace webpage
83
+
# to associate page_id and service_id
84
+
download: true
85
+
path: ./page_map
70
86
71
-
9. Run from terminal: `./rsmetrics.py` to run RSmetrics
87
+
export: true
88
+
from: 'page_map' # or 'source'
89
+
90
+
# Calculate source's metrics
91
+
Metrics: true
92
+
93
+
```
94
+
95
+
9. Run from terminal: `./rsmetrics.py --users --services` to run RSmetrics and include the `users.csv` and `services.csv` files generated by the Preprocessor
The `webservice` folder hosts a simple webservice implemented in Flask framework which can be used to host the report results.
181
+
182
+
__Note__: Please make sure you work in a virtual environment and you have already downloaded the required dependencies by issuing
183
+
`pip install -r requirements.txt`
184
+
185
+
The webservice application serves two endpoints
186
+
- `/` : This is the frontend webpage that displays the Report Results in a UI
187
+
- `/api` : This api call returns the evaluation metrics in json format
188
+
189
+
To run the webservice issue:
190
+
```
191
+
cd ./webservice
192
+
flask run
193
+
```
194
+
195
+
The webservice by default runs in localhost:5000 you can override this by issuing for example:
196
+
```
197
+
flask run -h 127.0.0.1 -p 8080
198
+
```
199
+
200
+
There is an env variable `RS_EVAL_METRIC_SOURCE` which directs the webservice to the generated `metrics.json` file produced after the evaluation process.
201
+
This by default honors this repo's folder structure and directs to the root `/data/metrics.json` path
202
+
203
+
You can override this by editing the `.env` file inside the `/webservice` folder, or specificy the `RS_EVAL_METRIC_SOURCE` variable accordingly before executing the `flask run`command
The number of user clicks through recommendations panels divided by the total times recommendation panels were presented to users.
5
+
6
+
description: >
7
+
The number of user clicks through recommendations panels divided by the total times recommendation panels were presented to users. Takes into account all historical data of user actions. The metric is expressed by the formula: $$Click-Through Rate=\frac{clicks}{views}$$
8
+
output:
9
+
type: float
10
+
min: 0
11
+
max: +inf
12
+
comment: A value of 0 indicates that no clicks through recommendations panels occurred
13
+
14
+
prerequisites:
15
+
- all available user actions
16
+
17
+
process:
18
+
- step: Retrieve user actions with recommendation panel
19
+
details: >
20
+
Get only the user actions that present a recommendation panel to the user in the source page. Those are actions with the following source paths: (i) /services, (ii) /services/, (iii) /services/c/{any category name}
21
+
- step: Count user actions with recommendation panel
22
+
details: >
23
+
Count the items in the above list as they represent the times recommendations panels were presented to the users of the portal
24
+
- step: Filter list
25
+
details: >
26
+
Narrow the above list into a new subset by selecting only user actions that originate from a recommendation panel. Those are actions that have the 'recommendation' string in the Action column
27
+
- step: Count user actions with clicks through recommendation panel
28
+
details: >
29
+
Count the items in the subset as they represent the times users clicked through recommendations
30
+
- step: Calculate ratio
31
+
details: >
32
+
Divide the items of the subset with the items of the first list to get the click-through rate
33
+
34
+
# This is optional for visual stylization of the metric when displayed on the report
Measures Recommendations' diversity. The index is 0 when all items are chosen equally often, and 1 when a single item is always chosen.
5
+
6
+
description: >
7
+
The diversity (\(G\)) of the recommendations according to Gini Index. The index is 0 when all items are chosen equally often,
8
+
and 1 when a single item is always chosen
9
+
(see book \(\href{https://link.springer.com/10.1007/978-1-4939-7131-2_110158}{https://link.springer.com/10.1007/978-1-4939-7131-2_110158}\)). Generally, the Gini Index mathematical expression is defined as:
10
+
$$G=\frac{1}{n-1}\sum_{j=1}^{n}(2j-n-1)p(i_j)$$where \(i_1,\ldots,i_n\) is the list of items ordered according to increasing \(p(i)\) and each item \(i\) accounts for a proportion \(p(i)\) of user recommendations. In RS Metrics the computation is determined by the following forumla:
comment: The index is 0 when all items are chosen equally often, and 1 when a single item is always chosen.
18
+
19
+
prerequisites:
20
+
- recommendations without anonymous users
21
+
- all available services
22
+
23
+
process:
24
+
- step: Clean up
25
+
details: >
26
+
Recommendations clean up; entries removal where users or services are not found in "users" or "services" files accordingly
27
+
- step: Services Impact
28
+
details: >
29
+
Calculation of the impact of the services, by counting how many times each service i was suggested to all possible users: count(j)
30
+
- step: Sort Services Impact from low to high
31
+
details: >
32
+
Sort the number of how many times each service (i.e. i) was suggested from the lower to the higher value, in order to apply the respective weight (j). The computation includes services with 0 recommendation occurrence
33
+
- step: Recommended Probability of the Services
34
+
details: >
35
+
For each service calculate its recommended probability by dividing the number of service's occurrence found in the recommendations to the total number of recommendations
36
+
- step: Service-based product computation
37
+
details: >
38
+
Calculation of the product of the recommended probability from previous step and services' respective index j, for each service individually
39
+
- step: Gini Index computation
40
+
details: >
41
+
Computation of the overall value by summing all values from previous step
42
+
43
+
# This is optional for visual stylization of the metric when displayed on the report
Measures Recommendations' diversity. The entropy is 0 when a single item is always chosen or recommended,
5
+
and log n when n items are chosen or recommended equally often.
6
+
7
+
description: >
8
+
The diversity (\(H\)) of the recommendations according to Shannon Entropy. The entropy is 0 when a single item
9
+
is always chosen or recommended, and log(n) when n items are chosen or recommended equally often
10
+
(see book \(\href{https://link.springer.com/10.1007/978-1-4939-7131-2_110158}{https://link.springer.com/10.1007/978-1-4939-7131-2_110158}\)). Generally, the Shannon Entropy mathematical expression is defined as:
11
+
$$H=-\sum_{i=1}^{n}p(i)\log_2 p(i) $$In RS Metrics the computation is determined by the following forumla:
comment: The entropy is 0 when a single item is always chosen or recommended, and log n when n items are chosen or recommended equally often.
19
+
20
+
prerequisites:
21
+
- recommendations without anonymous users
22
+
- all available services
23
+
24
+
process:
25
+
- step: Clean up
26
+
details: >
27
+
Recommendations clean up; entries removal where users or services are not found in "users" or "services" files accordingly
28
+
- step: Services Impact
29
+
details: >
30
+
Calculation of the impact of the services, by counting how many times each service i was suggested to all possible users: count(i)
31
+
- step: Recommended Probability of the Services
32
+
details: >
33
+
For each service calculate its recommended probability by dividing the number of service's occurrences found in the recommendations to the total number of recommendations
34
+
- step: Service-based product computation
35
+
details: >
36
+
Calculation of the product of the recommended probability from previous step and the logarithmic value of it, for each service individually
37
+
- step: Shannon Entropy computation
38
+
details: >
39
+
Computation of the overall value by summing all values from previous step
40
+
41
+
# This is optional for visual stylization of the metric when displayed on the report
0 commit comments