[BUG]: AzureFileCopyV5: Incorrectly returns "Storage account not found" error for recently created Storage account.

[mm2709]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

AzureFileCopy

Task version

V5

Issue Description

In my pipeline, I'm creating a new storage account. Then in the immediate next step, I'm trying to copy files to this newly created storage account. However, I'm receiving the error "Storage account: not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only."

It seems the highlighted call to "Get-AzResource" cmdlet is returning nothing/null and eventually ends up throwing the exception.

I confirmed the above behavior by running the same cmdlet from my dev machine. Get-AzResource cmdlet returns nothing/null, but Get-AzStorageAccount cmdlet returns the storage account details and confirms its existence.

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Microsoft Windows Server 2022

Relevant log output

2024-02-21T21:23:06.8331198Z ##[debug][Azure Call]Retrieved resource details successfully for azure storage account resource: ugbmriexfyge4 with resource type: Microsoft.Storage/storageAccounts
2024-02-21T21:23:06.8417318Z ##[debug](ARM)Storage account: ugbmriexfyge4 not found
2024-02-21T21:23:06.8799015Z ##[debug]Processed: ##vso[task.logissue type=error;code={"Task_Internal_Error":"RMStorageAccountNotFound"};]
2024-02-21T21:23:06.9549946Z ##[debug]System.Management.Automation.RuntimeException: Storage account: ugbmriexfyge4 not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.
2024-02-21T21:23:06.9571527Z ##[debug]Processed: ##vso[task.logissue type=error;code={"Task_Internal_Error":"TemporaryCopyingToBlobContainerFailed"};]
2024-02-21T21:23:06.9617576Z ##[debug]Trying to disconnect from Azure and clear context at process scope
2024-02-21T21:23:06.9703679Z ##[debug]Cannot verify the Microsoft .NET Framework version 4.7.2 because it is not included in the list of permitted versions.
2024-02-21T21:23:06.9732368Z ##[debug]Populating RepositorySourceLocation property for module Az.Accounts.
2024-02-21T21:23:06.9760483Z ##[debug]Loading module from path 'C:\Modules\az_9.3.0\Az.Accounts\2.15.1\Az.Accounts.psm1'.
2024-02-21T21:23:07.0482685Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop
2024-02-21T21:23:07.0858552Z ##[command]Clear-AzContext -Scope Process -ErrorAction Stop
2024-02-21T21:23:07.1376557Z ##[debug]Caught exception from task script.
2024-02-21T21:23:07.1418622Z ##[debug]Error record:
2024-02-21T21:23:07.2000401Z ##[debug]Storage account: ugbmriexfyge4 not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.
2024-02-21T21:23:07.2015479Z ##[debug]At D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\5.234.0\AzureUtilityAz1.0.ps1:22 char:13
2024-02-21T21:23:07.2031483Z ##[debug]+             Throw (Get-VstsLocString -Key "AFC_StorageAccountNotFound ...
2024-02-21T21:23:07.2047633Z ##[debug]+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2024-02-21T21:23:07.2062023Z ##[debug]    + CategoryInfo          : OperationStopped: (Storage account...ager type only.:String) [], RuntimeException
2024-02-21T21:23:07.2075945Z ##[debug]    + FullyQualifiedErrorId : Storage account: ugbmriexfyge4 not found. The selected service connection 'Service Princ    ipal' supports storage accounts of Azure Resource Manager type only.
2024-02-21T21:23:07.2089757Z ##[debug] 
2024-02-21T21:23:07.2110696Z ##[debug]Script stack trace:
2024-02-21T21:23:07.2150702Z ##[debug]at Get-AzureStorageAccountResourceGroupName, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\5.234.0\AzureUtilityAz1.0.ps1: line 22
2024-02-21T21:23:07.2174345Z ##[debug]at Get-AzureStorageKeyFromARM, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\5.234.0\AzureUtilityRest.ps1: line 13
2024-02-21T21:23:07.2188773Z ##[debug]at Get-StorageKey, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\5.234.0\Utility.ps1: line 72
2024-02-21T21:23:07.2203930Z ##[debug]at <ScriptBlock>, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\5.234.0\AzureFileCopy.ps1: line 125
2024-02-21T21:23:07.2218571Z ##[debug]at <ScriptBlock>, <No file>: line 1
2024-02-21T21:23:07.2232900Z ##[debug]at <ScriptBlock>, <No file>: line 22
2024-02-21T21:23:07.2246853Z ##[debug]at <ScriptBlock>, <No file>: line 18
2024-02-21T21:23:07.2262865Z ##[debug]at <ScriptBlock>, <No file>: line 1
2024-02-21T21:23:07.2287128Z ##[debug]Exception:
2024-02-21T21:23:07.2308201Z ##[debug]System.Management.Automation.RuntimeException: Storage account: ugbmriexfyge4 not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.
2024-02-21T21:23:07.2372367Z ##[error]Storage account: ugbmriexfyge4 not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.
2024-02-21T21:23:07.2373535Z ##[debug]Processed: ##vso[task.logissue type=error]Storage account: ugbmriexfyge4 not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.
2024-02-21T21:23:07.2388283Z ##[debug]Processed: ##vso[task.complete result=Failed]
2024-02-21T21:23:07.3037523Z ##[section]Finishing: Copy Templates v5

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION] 

Repro steps

Step 1: Create a new storage account
Step 2: Use AzureFileCopyv5 task to copy files to the above storage account

[v-mohithgc]

Hi @mm2709, is there any fix do you like to purpose here?

[mm2709]

Hi @mm2709, is there any fix do you like to purpose here?

Yes. I have the following options to fix the issue.

Option 1: I have already mentioned it in the description that using Get-AzStorageAccount cmdlet instead of Get-AzResource should fix the issue. All we have to do is replace

$azureStorageAccountResourceDetails = (Get-AzResource -ErrorAction Stop) | Where-Object { ($_.ResourceType -eq $ARMStorageAccountResourceType) -and ($_.Name -eq $storageAccountName)}
with

$azureStorageAccountResourceDetails = (Get-AzStorageAccount -ErrorAction Stop) | Where-Object {($_.StorageAccountName -eq $storageAccountName)}

Option 2: We can supply the ResourceGroupName (keep it as optional for backward compatibility) as input parameter from the pipeline task. Then we can skip the call Get-AzureStorageAccountResourceGroupName (in below screenshot) if the ResourceGroupName is not null or empty.

[pdtit]

having the exact same issue since a few days, but not constant. 1 out of 5 pipeline runs still get through. Had the same with azurefilecopy@4 lately, and started integrating a 5min wait task before creating the storage account and running azcopy. To no avail. The storage account and container is getting created fine, but the task doesn't see it...

[ybadragon]

We recently started having this issue very frequently now. It started for us a couple days ago but would fix itself after about 5 minutes, today it is taking much longer to fix itself. This is occuring even on storage accounts that are brand new and have never been deleted/re-created

[wjdavis5]

I'm also running into this on the windows-2022 agent in ADO. I create a storage account and a few steps later in the pipeline try to copy a file to it and it fails.

[rkistinger-carvana]

I'm also running into this on the windows-2022 agent in ADO. I create a storage account and a few steps later in the pipeline try to copy a file to it and it fails.

I am experiencing the same issue with this agent.

[robdlee]

Also happening to me too

[mm2709]

Hi @mm2709, is there any fix do you like to purpose here?

Yes. I have the following options to fix the issue.

Option 1: I have already mentioned it in the description that using Get-AzStorageAccount cmdlet instead of Get-AzResource should fix the issue. All we have to do is replace

$azureStorageAccountResourceDetails = (Get-AzResource -ErrorAction Stop) | Where-Object { ($_.ResourceType -eq $ARMStorageAccountResourceType) -and ($_.Name -eq $storageAccountName)} with

$azureStorageAccountResourceDetails = (Get-AzStorageAccount -ErrorAction Stop) | Where-Object {($_.StorageAccountName -eq $storageAccountName)}

Option 2: We can supply the ResourceGroupName (keep it as optional for backward compatibility) as input parameter from the pipeline task. Then we can skip the call Get-AzureStorageAccountResourceGroupName (in below screenshot) if the ResourceGroupName is not null or empty.

@v-mohithgc Option 1 is a straight-forward fix. Can you please prioritize and assign this bug?