Unable to push cache files to remote S3 storage

[shamilsaeed]

Hi,

I am running 'dvc repro' and made changes to dependencies files, it runs the dvc stage of pipeline. This stage only trains the model, and the output is just a model.tar.gz file. This model.tar.gz file is included in dvc.yml file as an outputs.
The issue is no matter what i do, i never see the output cache ever getting uploaded in my s3 bucket. The path is correct, but the cache directory in s3 is empty. When i run pipeline, this is what i get:

Use dvc push to send your updates to remote storage.
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: s3://......../something/model.tar.gz

What am I doing wrong? I did 'dvc push --remote mys3cache' but still same message.

I use:
dvc[all]==2.23.0

[daavoo]

Hi @shamilsaeed , could you please:

• Upgrade DVC with pip install dvc[s3] --upgrade
• Share the output of dvc push -vv
• Share your dvc.yaml (remove any confidential info you might not want to share)
• Share the output of dvc config --list --show-origin (remove any confidential info you might not want to share)

[shamilsaeed]

Hi @daavoo ,

Here is the changes as per your mentioned. I masked some information. I did dvc repro --dry to bypass the actually training step but still generate remote cache, hoping that should be ok.

• Added dvc[s3] in my requirements.txt file, so it installed latest version
• Here is the output of 'dvc push -r s3cache -vv':

python step_1_submit_training.py
Use dvc push to send your updates to remote storage.
2023-03-23 16:36:55,679 TRACE: Namespace(all_branches=False, all_commits=False, all_tags=False, cd='.', cmd='push', cprofile=False, cprofile_dump=None, func=<class 'dvc.commands.data_sync.CmdDataPush'>, glob=False, instrument=False, instrument_open=False, jobs=None, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False), pdb=False, quiet=0, recursive=False, remote='s3cache', run_cache=False, targets=[], verbose=2, version=None, viztracer=False, viztracer_depth=None, with_deps=False, yappi=False)
2023-03-23 16:36:56,006 TRACE: params.yaml does not exist, it won't be used in parametrization
2023-03-23 16:36:56,006 TRACE: Context during resolution of stage step_1_submit_sagemaker_training:
{}
2023-03-23 16:36:56,273 TRACE: 296.58 ms in collecting stages from /home/vsts/work/1/s/folder
2023-03-23 16:36:56,274 TRACE: 9.00 mks in collecting stages from /home/vsts/work/1/s/folder/deploymentScripts
2023-03-23 16:36:56,275 TRACE: 377.80 mks in collecting stages from /home/vsts/work/1/s/folder/src
2023-03-23 16:36:56,275 TRACE: 11.20 mks in collecting stages from /home/vsts/work/1/s/folder/src/code
2023-03-23 16:36:56,276 TRACE: 7.00 mks in collecting stages from /home/vsts/work/1/s/folder/dvc_scripts
2023-03-23 16:36:56,292 DEBUG: Preparing to transfer data from '/home/vsts/work/1/s/folder/.dvc/cache' to 'mybucket/somefolder/new_cache/'
2023-03-23 16:36:56,292 DEBUG: Preparing to collect status from 'mybucket/somefolder/new_cache/'
2023-03-23 16:36:56,293 DEBUG: Collecting status from 'mybucket/somefolder/new_cache/'
2023-03-23 16:36:56,294 DEBUG: Querying 1 oids via object_exists
2023-03-23 16:36:56,738 DEBUG: Preparing to collect status from '/home/vsts/work/1/s/folder/.dvc/cache'
2023-03-23 16:36:56,739 DEBUG: Collecting status from '/home/vsts/work/1/s/model/.dvc/cache'
2023-03-23 16:36:56,741 WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: s3://mybucket/somefolder/output/model.tar.gz, etag: xxxxxx
Everything is up to date.
2023-03-23 16:36:56,745 DEBUG: Analytics is enabled.
2023-03-23 16:36:56,836 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmprrl0ea36']'
2023-03-23 16:36:56,839 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmprrl0ea36']'

• here is my dvc.yaml:

stages:
step_1_submit_training:
cmd: python step_1_submit_training.py --s3_bucket_ca=xx --s3_job_folder=xx
--s3_input_data_folder=xx --s3_output_model_filename_ca=somefolder/output/model.tar.gz
--sagemaker_region=xxxx
deps:
- dvc_scripts/step_1_submit_training.py
- s3://mybucket/somefolder/input_training_data
- src/train.py
outs:
- s3://mybucket/somefolder/output/model.tar.gz

• output of dvc config --list --show-origin:

.dvc/config remote.s3cache.url=s3://mybucket/somefolder/new_cache/
.dvc/config cache.s3=s3cache

[daavoo]

Thanks for the details! I will take a look

[shamilsaeed]

@daavoo ,

I was able to push the output i.e s3://mybucket/somefolder/output/model.tar.gz by doing git add s3://mybucket/somefolder/output/model.tar.gz -r s3cache --to-remote.

I can see files in s3 remote cache now, but regardless of that, the training stage always kicks off if i rerun pipeline. If i remove the following line in dvc.yml, then it works as expected by not rerunning the same training stage. The problem is solely to do with the output:

outs:
-s3://mybucket/somefolder/output/model.tar.gz

I might just end up removing this outputs from dvc.yml, as it is not critical to versioning model. However, i am still curious as to why dvc.lock, remote cache fails with the outputs. Looking forward to hearing.