Skip to content

dvc exp show: External s3 address not properly shown #9757

Open
@kpetersen-hf

Description

@kpetersen-hf

Bug Report

Description

Hello,

I extended the example from #9713. Thank you so much for addressing that so quickly! This is much appreciated!

When now using an external s3 address s3://<BUCKET>/<FILE_NAME> (e.g., s3://my_bucket/model.pkl) as an output location in DVC 3.7, workspace and master branch in dvc exp show use two different names to refer to the s3 location, neither of which seems correct: master uses <REPO_PATH>/<BUCKET>/<FILE_NAME>, workspace uses <BUCKET>/<FILENAME>. Both are missing the prefix s3://

Reproduce

For reproducing, please specify the respective <BUCKET> and <FILE_NAME> in the following:

git init -q dvc_issue
cd dvc_issue
dvc init -q

cat <<EOT >> .dvc/config
[cache]
    type = symlink
EOT

cat <<EOT >> dvc.yaml
vars:
  - uri_model: s3://<BUCKET>/<FILE_NAME>
stages:
  train:
    cmd: python train.py
    deps:
    - train.py
    outs:
    - \${uri_model}:
        cache: false
  evaluate:
    cmd: python evaluate.py
    deps:
    - evaluate.py
    - \${uri_model}
    metrics:
      - metrics.json:
          cache: false
EOT

cat <<EOT >> train.py
import boto3
def main():
    bucket_name = <BUCKET>
    file_name = <FILE_NAME>
    data = b"weights: 1, 2, 3"
    s3 = boto3.resource('s3')
    object = s3.Object(bucket_name, file_name)
    object.put(Body=data)
    print("Finished train.")
if __name__ == "__main__":
    main()
EOT

cat <<EOT >> evaluate.py
import json
def main():
    metrics_filename = "metrics.json"
    data = {"auc": 0.29}
    with open(metrics_filename, 'w') as f:
        json.dump(data, f)
    print("Finished evaluate.")
if __name__ == "__main__":
    main()
EOT

dvc repro -q
git add .
git commit -q -m "initial"
dvc exp show -v

Expected

A single column with the entry s3://<BUCKET>/<FILENAME>.

Environment information

Output of dvc doctor:

$ dvc doctor
VC version: 3.7.0 (pip)
------------------------
Platform: Python 3.10.8 on Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-glibc2.17
Subprojects:
	dvc_data = 2.6.0
	dvc_objects = 0.23.1
	dvc_render = 0.5.3
	dvc_task = 0.3.0
	scmrepo = 1.0.4
Supports:
	http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.6.0, boto3 = 1.26.0)
Config:
	Global: /home/kpetersen/.config/dvc
	System: /etc/xdg/dvc

-->

Metadata

Metadata

Assignees

No one assigned

    Labels

    A: experimentsRelated to dvc expbugDid we break something?p2-mediumMedium priority, should be done, but less importantuiuser interface / interaction

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions