Open
Description
Bug Report
Description
Hello,
I extended the example from #9713. Thank you so much for addressing that so quickly! This is much appreciated!
When now using an external s3 address s3://<BUCKET>/<FILE_NAME>
(e.g., s3://my_bucket/model.pkl
) as an output location in DVC 3.7, workspace
and master
branch in dvc exp show
use two different names to refer to the s3 location, neither of which seems correct: master
uses <REPO_PATH>/<BUCKET>/<FILE_NAME>
, workspace
uses <BUCKET>/<FILENAME>
. Both are missing the prefix s3://
Reproduce
For reproducing, please specify the respective <BUCKET>
and <FILE_NAME>
in the following:
git init -q dvc_issue
cd dvc_issue
dvc init -q
cat <<EOT >> .dvc/config
[cache]
type = symlink
EOT
cat <<EOT >> dvc.yaml
vars:
- uri_model: s3://<BUCKET>/<FILE_NAME>
stages:
train:
cmd: python train.py
deps:
- train.py
outs:
- \${uri_model}:
cache: false
evaluate:
cmd: python evaluate.py
deps:
- evaluate.py
- \${uri_model}
metrics:
- metrics.json:
cache: false
EOT
cat <<EOT >> train.py
import boto3
def main():
bucket_name = <BUCKET>
file_name = <FILE_NAME>
data = b"weights: 1, 2, 3"
s3 = boto3.resource('s3')
object = s3.Object(bucket_name, file_name)
object.put(Body=data)
print("Finished train.")
if __name__ == "__main__":
main()
EOT
cat <<EOT >> evaluate.py
import json
def main():
metrics_filename = "metrics.json"
data = {"auc": 0.29}
with open(metrics_filename, 'w') as f:
json.dump(data, f)
print("Finished evaluate.")
if __name__ == "__main__":
main()
EOT
dvc repro -q
git add .
git commit -q -m "initial"
dvc exp show -v
Expected
A single column with the entry s3://<BUCKET>/<FILENAME>
.
Environment information
Output of dvc doctor
:
$ dvc doctor
VC version: 3.7.0 (pip)
------------------------
Platform: Python 3.10.8 on Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-glibc2.17
Subprojects:
dvc_data = 2.6.0
dvc_objects = 0.23.1
dvc_render = 0.5.3
dvc_task = 0.3.0
scmrepo = 1.0.4
Supports:
http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
s3 (s3fs = 2023.6.0, boto3 = 1.26.0)
Config:
Global: /home/kpetersen/.config/dvc
System: /etc/xdg/dvc
-->