azcopy --preserve-symlinks returning error code if symlinks target is not resolved

[dwr-psandhu]

Which version of the AzCopy was used?

10.19.0

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy copy "http...?$SAS" localdir --recursive --preserve-symlinks

What problem was encountered?

All symlinks and files were copied to local. However the return code from the command is non-zero resulting in the bash script with set -e

How can we reproduce the problem in the simplest way?

Run the command check exit code

Have you found a mitigation/solution?

ignoring the error temporarily
cmd || true

[gapra-msft]

@dwr-psandhu could you please share the azcopy log file with log-level set to DEBUG?

[dwr-psandhu]

@gapra-msft Not exactly what you want but this is what I have for now. I have since backed away from azcopy to using rsync with mounted blobs

2023/07/05 22:45:31 ERR: [P#0-T#40] DOWNLOADFAILED: https://xxx.blob.core.windows.net/xxx/simulations/2016_base/sflux/sflux_air_1.0133.nc?se=2023-07-12T22%3A38Z&sig=-REDACTED-&sp=racwdl&sr=c&sv=2021-06-08 : 000 : symlink ../../../data/atmos/baydelta_sflux_v20220916/baydelta_schism_air_20160906.nc /mnt/resource/batch/tasks/workitems/2016_base_baroclinic_job_20230705153810/job-1/2016_base_baroclinic_task_20230705153810/wd/simulations/simulations/2016_base/sflux/sflux_air_1.0133.nc: no such file or directory. When creating destination symlink (check operation type, is it really download?). X-Ms-Request-Id: 

   Dst: /mnt/resource/batch/tasks/workitems/2016_base_baroclinic_job_20230705153810/job-1/2016_base_baroclinic_task_20230705153810/wd/simulations/simulations/2016_base/sflux/sflux_air_1.0133.nc
2023/07/05 22:45:31 ==> REQUEST/RESPONSE (Try=1/7.502154ms, OpTime=29.589791ms) -- RESPONSE SUCCESSFULLY RECEIVED

[gapra-msft]

@dwr-psandhu Do you have the X-Ms-Request-Id: at the end? Could you paste that here please?

[dwr-psandhu]

Sorry about that. Here is a more complete request/response for the particular symbolic link.

2023/07/05 22:45:31 ==> REQUEST/RESPONSE (Try=1/6.420088ms, OpTime=27.865286ms) -- RESPONSE SUCCESSFULLY RECEIVED
   GET https://xxx.blob.core.windows.net/itp202306/simulations/2016_base/sflux/sflux_air_1.0133.nc?se=2023-07-12T22%3A38Z&sig=-REDACTED-&sp=racwdl&sr=c&sv=2021-06-08&timeout=901
   X-Ms-Request-Id: [7442e7c7-601e-0106-2392-af9ea7000000]

2023/07/05 22:45:31 ==> REQUEST/RESPONSE (Try=1/7.729368ms, OpTime=35.244533ms) -- RESPONSE SUCCESSFULLY RECEIVED
   GET https://xxx.blob.core.windows.net/itp202306/simulations/2016_base/sflux/sflux_prc_1.0216.nc?se=2023-07-12T22%3A38Z&sig=-REDACTED-&sp=racwdl&sr=c&sv=2021-06-08&timeout=901
   X-Ms-Request-Id: [c64a128d-c01e-0072-4992-af545a000000]

2023/07/05 22:45:31 ERR: [P#0-T#40] DOWNLOADFAILED: https://xxx.blob.core.windows.net/itp202306/simulations/2016_base/sflux/sflux_air_1.0133.nc?se=2023-07-12T22%3A38Z&sig=-REDACTED-&sp=racwdl&sr=c&sv=2021-06-08 : 000 : symlink ../../../data/atmos/baydelta_sflux_v20220916/baydelta_schism_air_20160906.nc /mnt/resource/batch/tasks/workitems/2016_base_baroclinic_job_20230705153810/job-1/2016_base_baroclinic_task_20230705153810/wd/simulations/simulations/2016_base/sflux/sflux_air_1.0133.nc: no such file or directory. When creating destination symlink (check operation type, is it really download?). X-Ms-Request-Id: 

   Dst: /mnt/resource/batch/tasks/workitems/2016_base_baroclinic_job_20230705153810/job-1/2016_base_baroclinic_task_20230705153810/wd/simulations/simulations/2016_base/sflux/sflux_air_1.0133.nc

The symbolic link points to a directory outside the one being created and the files pointed to have not yet been copied over. So the file the symbolic link does not exist for now.

[adreed-msft]

Interesting-- In our testing, symlinks could be targeting objects that simply do not exist and it would work out fine. This'll need an investigation.

[dwr-psandhu]

@adreed-msft I have moved to using rsync from the same mounted blob to the local storage and that has no issues with copying the symlinks as expected. Let me know if you need more information about this particular blob symlink.

blobfuse2 mounting was used when setting up the symlinks from a linux vm, however blobfuse (v1) is what Azure batch uses when using mount configuration options which was used for the job. Both seem to work just fine.

[cathalobrien]

Hello, I am experiencing the same error as reported here but my symlinks aren't even being copied to local. @adreed-msft did your investigation turn anything up?

Which version of the AzCopy was used?

10.19.0

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy copy "http...?$SAS" . --recursive --preserve-symlinks

What problem was encountered?

The download of symlinks failed, see error code below. Unlike the person above the symlinks did not even appear on my local machine. Strangely, I can download them from the portal successfully

error message:

2023/08/10 11:58:59 ==> REQUEST/RESPONSE (Try=1/10.259455ms, OpTime=25.524736ms) -- RESPONSE SUCCESSFULLY RECEIVED
   GET https://<container>?<SAS>
   X-Ms-Request-Id: [bd58cec4-501e-0011-1582-cb3689000000]

2023/08/10 11:58:59 ERR: [P#0-T#0] DOWNLOADFAILED: https://<Container>?<SAS> : 000 : symlink ../full_data_eps /shared/home/hpc_admin/ppt00000600/: no such file or directory. When creating destination symlink (check operation type, is it really download?). X-Ms-Request-Id:

   Dst: /shared/home/hpc_admin/ppt00000600/

How can we reproduce the problem in the simplest way?

Upload a symlink to blob storage (I used --preserve-symlinks on upload). Then try download with the command above

Have you found a mitigation/solution?

sadly not

[dwr-psandhu]

This is still happening with azcopy version 10.24.0. If the symlink destination cannot be resolved it should simply create the link. This is congrous with the behavior of ln -s which can create symbolic links to targets not yet created.