Pipeline getting failed sometimes

[sohamkyagit]

logs_4250322 --successfull.zip
logs_4250360 --failed.zip

Hello,
I have shared you the details of some of the logs of the pipelines, which are runs on a daily basis but its was getting failed frequently. Same pipeline if successful today is may failed tomorrow. We didn't do any changes for the pipeline. I have attached the logs for the pipelines can you please look into this on a urgent basis. As this is the production issue.

Thanks,
Soham Rakshit.

[devaroqu]

Hello Team,

adding more info and details on the issue as I have been working with customer
customer has self host vm using devops pipeline to download files from storage account
it start by creating the environment ubuntu-22.04
2024-11-06T04:30:23.5046474Z Agent machine name: 'fv-az419-91'
2024-11-06T04:30:23.5046705Z Current agent version: '3.246.0'
2024-11-06T04:30:23.5083175Z ##[group]Operating System
2024-11-06T04:30:23.5083437Z Ubuntu
2024-11-06T04:30:23.5083577Z 22.04.5
2024-11-06T04:30:23.5083696Z LTS
2024-11-06T04:30:23.5083841Z ##[endgroup]
2024-11-06T04:30:23.5083988Z ##[group]Runner Image
2024-11-06T04:30:23.5084170Z Image: ubuntu-22.04
2024-11-06T04:30:23.5084345Z Version: 20241015.1.0

the pipeline jobs download files from storage account and here we are facing the issue we intermittently get 403 while we do the downloads although we whitelist the IP address of the devops agent in the storage account
one day the job would work the other day it will fail to download the file

azcopy version 10.26

2024-11-06T04:36:04.5284500Z + azcopy sync 'https://braunusstagingproducts.blob.core.windows.net/braunstagingproductsdetails?sp=' /home/vsts/work/1/s/ --recursive=true

2024-11-06T04:36:04.5285178Z /home/vsts/work/1/s
2024-11-06T04:36:04.5385782Z INFO: azcopy 10.26.0: A newer version 10.27.0 is available to download
2024-11-06T04:36:04.5386199Z
2024-11-06T04:36:04.5485692Z INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2024-11-06T04:36:04.5486128Z
2024-11-06T04:36:04.5486751Z Job c0a159bc-b494-8147-5aeb-e8c6c1e2f8a9 has started
2024-11-06T04:36:04.5489134Z Log file is located at: /home/vsts/.azcopy/c0a159bc-b494-8147-5aeb-e8c6c1e2f8a9.log
2024-11-06T04:36:04.5489420Z
2024-11-06T04:36:04.6177275Z
2024-11-06T04:36:04.6179704Z Cannot perform sync due to error: cannot list files due to reason GET https://braunusstagingproducts.blob.core.windows.net/braunstagingproductsdetails
2024-11-06T04:36:04.6180459Z --------------------------------------------------------------------------------
2024-11-06T04:36:04.6180758Z RESPONSE 403: 403 This request is not authorized to perform this operation.
2024-11-06T04:36:04.6181068Z ERROR CODE: AuthorizationFailure
2024-11-06T04:36:04.6181416Z --------------------------------------------------------------------------------
2024-11-06T04:36:04.6181996Z AuthorizationFailureThis request is not authorized to perform this operation.
2024-11-06T04:36:04.6182429Z RequestId:7dd99ba4-c01e-0085-3d05-309690000000
2024-11-06T04:36:04.6182757Z Time:2024-11-06T04:36:04.6126096Z

comparing working and non working scenarios we see that using azcopy version 10.27 we dont see the 403 error

one thing to add is that in the pipeline we try to install the latest version of azcopy and we get 10.26 as seen below

downloading using this commands
echo 'Installing AzCopy'

Download and extract

wget https://aka.ms/downloadazcopy-v10-linux
tar -xvf downloadazcopy-v10-linux

Move AzCopy

sudo rm -f /usr/bin/azcopy
sudo cp ./azcopy_linux_amd64_*/azcopy /usr/bin/
sudo chmod 755 /usr/bin/azcopy

Clean the kitchen

rm -f downloadazcopy-v10-linux
rm -rf ./azcopy_linux_amd64_*/
azcopy --version

later we see

2024-11-06T04:30:49.2342636Z
2024-11-06T04:30:49.2375594Z azcopy_linux_amd64_10.27.0/
2024-11-06T04:30:49.2376276Z azcopy_linux_amd64_10.27.0/azcopy
2024-11-06T04:30:49.6384969Z azcopy_linux_amd64_10.27.0/NOTICE.txt
2024-11-06T04:30:51.4397599Z azcopy version 10.26.0
2024-11-06T04:30:51.8243414Z ##[debug]Agent environment resources - Disk: / Available 21287.02 MB out of 74244.74 MB, Memory: Used 961.00 MB out of 6921.00 MB, CPU: Usage 50.50%
2024-11-06T04:30:51.9627403Z INFO: azcopy 10.26.0: A newer version 10.27.0 is available to download
2024-11-06T04:30:51.9628017Z

can you help us investigate the inconsistency of the azcopy behavior?
when we try to download azcopy we sometime get 10.26 and sometimes 10.27

[gapra-msft]

@sohamkyagit these logs are from over a week ago (around the time when we were releasing 10.27), are you still experiencing this issue?

The latest version that should be downloaded consistently is 10.27.1

[sohamkyagit]

logs_4304700-latest-completed.zip
logs_4304740-latest-failed.zip
logs_4304800-latest-complete.zip

Hello, I have added todays log. Where with in 3 pipeline 1 got failed and 2 got completed I have attached these 3 logs.

Please let me know if anything is needed from my end.

Thanks,
Soham Rakshit

[devaroqu]

@gapra-msft good day!

have you had the chance to review the logs? I can still see the version of azcopy installed is old one

[sohamkyagit]

Hello Team,

Is their any update? I have attached the latest logs.

Thanks,
Soham Rakshit

[dphulkar-msft]

Hello,

I tried to download the latest version with the below commands. And it was showing the latest version.

wget https://aka.ms/downloadazcopy-v10-linux
tar -xvf downloadazcopy-v10-linux

Could you please confirm which azcopy binary is getting executed?

Command: which azcopy
This will output the full path of the binary being executed (e.g., /usr/bin/azcopy).

[sohamkyagit]

Hello team, Can you please share us your feasible time to connect? So we can connect over the call and sorted out these things. Please share us your comfortable time. Thanks & Regards, Soham Rakshit LTIMindtree Ltd Contractor working behalf of P&G Kolkata, India

…

________________________________ From: dphulkar-msft ***@***.***> Sent: 02 December 2024 17:11 To: Azure/azure-storage-azcopy ***@***.***> Cc: Soham Rakshit ***@***.***>; Mention ***@***.***> Subject: Re: [Azure/azure-storage-azcopy] Pipeline getting failed sometimes (Issue #2868) Caution - This email is from an external source. Please do not click on links or attachments if sender is unknown or from known person but the content is unusual. Never share your user ID or password under any circumstances. Hello, I tried to download the latest version with the below commands. And it was showing the latest version. wget https://aka.ms/downloadazcopy-v10-linux tar -xvf downloadazcopy-v10-linux Could you please confirm which azcopy binary is getting executed? Command: which azcopy This will output the full path of the binary being executed (e.g., /usr/bin/azcopy). — Reply to this email directly, view it on GitHub<#2868 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BIRZYVVPYWB7L6O45BG7NJ32DRBQNAVCNFSM6AAAAABR7FND62VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJRGMYDMMRUGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

________________________________ The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. LTIMindtree will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail.

[sohamkyagit]

Hello team, Can you please share us your feasible time to connect? So we can connect over the call and sorted out these things. Please share us your comfortable time.
Thanks,
Soham Rakshit

[dphulkar-msft]

Hello,

Could you try installing azcopy with the linux package manager and check once? If you still face the issues, we can connect over the call.

Commands to use:
sudo apt update
sudo apt install azcopy
azcopy --version

[derdanu]

Hello,

Could you try installing azcopy with the linux package manager and check once? If you still face the issues, we can connect over the call.

Commands to use: sudo apt update sudo apt install azcopy azcopy --version

More details can be found here: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10?tabs=dnf#install-azcopy-on-linux-by-using-a-package-manager

[sohamkyagit]

Hello team,

I need to connect with you. Can you please share your feasible time to connect? I will share you the meeting link.

Thanks,
Soham Rakshit.

[sohamkyagit]

Hello team,

I'm sharing you the teams meeting invitation to connect on this. Can you please try to join in this meeting and guide me to resolve this issue,

Meeting URL:
https://teams.microsoft.com/l/meetup-join/19%3ameeting_MzcxZDQzYjctNjVhMy00MjU5LThkYmItNTM4OWI5YWVhMTk4%40thread.v2/0?context=%7b%22Tid%22%3a%22ff355289-721e-4dd7-a663-afec62ab9d54%22%2c%22Oid%22%3a%22058c5537-d492-4c9f-bcf5-5a34754a945a%22%7d

please let me know if you are able to join this meeting or we can reschedule this meeting.

Thanks,
Soham Rakshit.

[sohamkyagit]

Hello team,

today we have updated as per your suggestion. We will see for 2 days and update you on the same.

Thanks,
Soham Rakshit