PR3: Update AWS dvc.yaml

README.md

@@ -45,16 +45,19 @@ for dataset in datasets:
 
 You can benchmark a group of supervised models:
 ```shell
-dvc repro benchmark/supervised/local/dvc.yaml
+dvc repro benchmark/supervised/local/dvc.yaml --single-item
 ```
 
 #### Zero-shot
 
 You can benchmark a group of zero-shot models:
 ```shell
-dvc repro benchmark/zero_shot/local/dvc.yaml
+dvc repro benchmark/zero_shot/local/dvc.yaml --single-item
 ```
 
+> [!NOTE]
+> Based on https://dvc.org/doc/command-reference/repro#-s, `--single-item` turns off the recursive search for all `dvc.yaml` changed dependencies. Only the current executed `dvc.yaml` will be searched.
+
 ### AWS environment
 
 There are two environments in which to run benchmark: one is the local environment, the other is the AWS environment.
@@ -84,12 +87,12 @@ The difference of the AWS environment is that:
 
 You can benchmark a group of supervised models:
 ```shell
-AWS_ACCOUNT_ID=xxx AWS_PROFILE=yyy dvc repro benchmark/supervised/aws/dvc.yaml
+AWS_ACCOUNT_ID=xxx AWS_PROFILE=yyy dvc repro benchmark/supervised/aws/dvc.yaml --single-item
 ```
 
 #### Zero-shot
 
 You can benchmark a group of zero-shot models:
 ```shell
-AWS_ACCOUNT_ID=xxx AWS_PROFILE=yyy dvc repro benchmark/zero_shot/aws/dvc.yaml
+AWS_ACCOUNT_ID=xxx AWS_PROFILE=yyy dvc repro benchmark/zero_shot/aws/dvc.yaml --single-item
 ```

benchmark/supervised/aws/params.yaml → benchmark/supervised/aws/default.yaml

@@ -7,13 +7,9 @@ aws:
   instance_type: "ml.m5.xlarge"
   volume_size: 5
 
-git:
-  git_cache_bust: 1
+output:
+  prediction: prediction
+  metric: metric
 
-source:
-  datasets_dir: ../../../datasets
-  models_dir: ../../../models
-
-destination:
-  output_dir: output
-  metric_dir: metric
+metrics: 
+  - spearman