[BUG]: Classic Release Pipelines Task Retries not working as expected

[AceOubahaTLI]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

ExtractFiles@1

Task version

1.*

Issue Description

Currently in Azure DevOps, we have a finnicky Windows Services deploy Classic Release stage, with a single Task Group that consists of 3 tasks:

1. Powershell script to stop the service using NET STOP, then wait for the associated PID to terminate (not manually terminating, just waiting)

• Runs every time

2. Extract Files stock Azure Devops task (ExtractFiles@1) to replace the files in the service directory

• Runs only if 1 succeeded

3. Powershell script to start the service using NET START

• Runs every time

Occasionally (less than 1% of the time), Task 2 will fail due to the files being locked (sometimes the service restarts to fast, or the OS doesn't unlock the files fast enough). A manual redeploy of the stage fixes the issue. I tried setting "Number of retries if task fails" under "Control Options" for task 2, with no luck (when task 2 errors out, task 3 starts right away, no retry occurs).

I have uploaded non-debug logs, since I cannot reliably enable logs on all of our release pipelines, nor can I reliably reproduce this issue. I have also attached a redacted version of the task definition, as my understanding is that all attachments are public.

What do I need to do to get this task/task group to automatically retry?

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

Version Dev19.M245.1 (AzureDevOps_M245_20241015.3)

Operation system

Windows 11

Relevant log output

2024-10-17T18:10:03.8209584Z ##[section]Finishing: Stop Service
2024-10-17T18:10:03.8263245Z ##[section]Starting: Extract files
2024-10-17T18:10:03.9313662Z ==============================================================================
2024-10-17T18:10:03.9313931Z Task : Extract files
2024-10-17T18:10:03.9314022Z Description : Extract a variety of archive and compression files such as .7z, .rar, .tar.gz, and .zip
2024-10-17T18:10:03.9314230Z Version : 1.246.3
2024-10-17T18:10:03.9314316Z Author : Microsoft Corporation
2024-10-17T18:10:03.9314417Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/extract-files
2024-10-17T18:10:03.9314595Z ==============================================================================
2024-10-17T18:10:04.6425903Z Searching for: [REDACTEDSERVICENAME]*.zip under directory: [REDACTEDAZUREAGENTWORKDIRECTORY]\r2\a_[REDACTEDSERVICENAME][REDACTEDSERVICENAME].zip
2024-10-17T18:10:04.6637884Z Found: 1 files to extract:
2024-10-17T18:10:04.6690098Z [REDACTEDAZUREAGENTWORKDIRECTORY]\r2\a_[REDACTEDSERVICENAME][REDACTEDSERVICENAME].zip[REDACTEDSERVICENAME].[REDACTEDVERSION].zip
2024-10-17T18:10:04.6690831Z Cleaning destination folder before extraction: [REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME]
2024-10-17T18:10:04.9313929Z C:[REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME][DLL1].dll - Access is denied.
2024-10-17T18:10:04.9316557Z [REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME][DLL2].dll - Access is denied.
2024-10-17T18:10:04.9338833Z [REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME][DLL3].dll - Access is denied.
2024-10-17T18:10:04.9339635Z [REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME][DLL4].dll - Access is denied.
2024-10-17T18:10:04.9340307Z [REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME][DLL5].dll - Access is denied.
2024-10-17T18:10:04.9356724Z [REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME][DLL6].dll - Access is denied.
2024-10-17T18:10:04.9357894Z [REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME][DLL7].dll - Access is denied.
2024-10-17T18:10:04.9358381Z [REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME][DLL8].dll - Access is denied.
2024-10-17T18:10:04.9359072Z [REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME][REDACTEDSERVICENAME].exe - Access is denied.
2024-10-17T18:10:04.9359972Z Error: Failed rmRF: EPERM: operation not permitted, unlink '[REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME]'
2024-10-17T18:10:04.9553372Z ##[error]Failed rmRF: EPERM: operation not permitted, unlink '[REDACTEDWINDOWSSERVICEDIRECTORY][REDACTEDSERVICENAME]'
2024-10-17T18:10:04.9933444Z ##[section]Finishing: Extract files
2024-10-17T18:10:04.9982682Z ##[section]Starting: Start Service

Full task logs with system.debug enabled

Unable to obtain debug logs since this failure happens on less than 1% of deploys, and is not reliably reproducible

Repro steps

{
"buildConfigMapping": {
},
"author": "",
"category": "Deploy",
"comment": "",
"createdBy": {
"displayName": "",
"id": "",
"uniqueName": ""
},
"createdOn": "2024-08-22T15:30:12.300Z",
"dataSourceBindings": [
],
"definitionType": "metaTask",
"demands": [
],
"description": "Deploy a service to a given path\n(Custom CompanyName Task)",
"execution": {
},
"friendlyName": "Deploy Service",
"groups": [
],
"iconUrl": "https://cdn.vsassets.io/v/M243_20240812.7/_content/icon-meta-task.png",
"id": "",
"inputs": [
{
"aliases": [
],
"defaultValue": "$(ServiceName)",
"groupName": "",
"helpMarkDown": "",
"label": "ArtifactName",
"name": "ArtifactName",
"options": {
},
"properties": {
},
"required": true,
"type": "string"
},
{
"aliases": [
],
"defaultValue": "$(PhysicalPath)",
"groupName": "",
"helpMarkDown": "",
"label": "PhysicalPath",
"name": "PhysicalPath",
"options": {
},
"properties": {
},
"required": true,
"type": "string"
},
{
"aliases": [
],
"defaultValue": "$(ServiceName)",
"groupName": "",
"helpMarkDown": "",
"label": "ServiceName",
"name": "ServiceName",
"options": {
},
"properties": {
},
"required": true,
"type": "string"
}
],
"instanceNameFormat": "Task group: Deploy Service $(ArtifactName)",
"modifiedBy": {
"displayName": "",
"id": "",
"uniqueName": ""
},
"modifiedOn": "2024-10-17T17:10:49.823Z",
"name": "Deploy Service",
"postJobExecution": {
},
"preJobExecution": {
},
"revision": 31,
"runsOn": [
"Agent",
"DeploymentGroup"
],
"satisfies": [
],
"sourceDefinitions": [
],
"tasks": [
{
"alwaysRun": false,
"condition": "succeeded()",
"continueOnError": false,
"displayName": "Stop Service",
"enabled": true,
"environment": {
},
"inputs": {
"arguments": "",
"debugPreference": "default",
"errorActionPreference": "continue",
"failOnStderr": "false",
"filePath": "",
"ignoreLASTEXITCODE": "false",
"informationPreference": "default",
"progressPreference": "silentlyContinue",
"pwsh": "false",
"runScriptInSeparateScope": "false",
"script": "whoami /user #used for debugging/troubleshooting\n\n$nid = Get-CimInstance -Class Win32_Service | Where-Object name -eq "$(ServiceName)" | Select-Object -ExpandProperty ProcessId\n\n\n$output = net stop $(ServiceName) 2>&1\nif(($LASTEXITCODE -eq 2) -and ($output | FIND /c "\"3521"") -gt 0){\necho "Service already stopped"\n$LASTEXITCODE = 0\n}\nelseif ($LASTEXITCODE -ne 0) { #If we had some other error\necho $output\n}\nelse {\necho $output\necho "Waiting for process $nid to stop..."\ndo {\n Start-Sleep -Seconds 1\n $process = get-process | Where-Object Id -eq $nid\n} while($process -ne $null)\necho "Process $nid stopped"\nStart-Sleep -Seconds 10 #let handles on directories and files unlock\n}",
"showWarnings": "false",
"targetType": "inline",
"verbosePreference": "default",
"warningPreference": "default",
"workingDirectory": ""
},
"retryCountOnTaskFailure": 0,
"task": {
"definitionType": "task",
"id": "",
"versionSpec": "2."
},
"timeoutInMinutes": 5
},
{
"alwaysRun": false,
"condition": "succeeded()",
"continueOnError": false,
"displayName": "Extract files ",
"enabled": true,
"environment": {
},
"inputs": {
"archiveFilePatterns": "$(System.ArtifactsDirectory)/$(ArtifactName)/$(ServiceName).zip/$(ServiceName).zip",
"cleanDestinationFolder": "true",
"destinationFolder": "$(PhysicalPath)\$(ServiceName)\",
"overwriteExistingFiles": "true",
"pathToSevenZipTool": ""
},
"retryCountOnTaskFailure": 3,
"task": {
"definitionType": "task",
"id": "",
"versionSpec": "1."
},
"timeoutInMinutes": 0
},
{
"alwaysRun": false,
"condition": "always()",
"continueOnError": false,
"displayName": "Start Service",
"enabled": true,
"environment": {
},
"inputs": {
"arguments": "",
"debugPreference": "default",
"errorActionPreference": "stop",
"failOnStderr": "false",
"filePath": "",
"ignoreLASTEXITCODE": "false",
"informationPreference": "default",
"progressPreference": "silentlyContinue",
"pwsh": "false",
"runScriptInSeparateScope": "false",
"script": "$output = net start $(ServiceName) 2>&1\nif(($LASTEXITCODE -eq 2) -and ($output | FIND /c "\"The requested service has already been started."") -gt 0){\necho "Service already started"\n$LASTEXITCODE = 0\n}\nelseif ($LASTEXITCODE -ne 0) { #If we had some other error\necho $output\n}",
"showWarnings": "false",
"targetType": "inline",
"verbosePreference": "default",
"warningPreference": "default",
"workingDirectory": ""
},
"retryCountOnTaskFailure": 0,
"task": {
"definitionType": "task",
"id": "",
"versionSpec": "2."
},
"timeoutInMinutes": 0
}
],
"version": {
"isTest": false,
"major": 1,
"minor": 0,
"patch": 0
}
}

[jacob0532]

Hi, I am currently having the same problem with my azure DevOps tasks. I did a test with a .ps1 file whereas script has an Exit 1 just to model the failure behavior, however, I don't notice retries from the task. I don't understand if this is due to a misconfiguration of the task or some other BUG.

I would appreciate if someone could help me with the same problem AceOubahaTLI is having.

YAML:

steps:

• powershell: |
  Exit 1

  failOnStderr: true
  displayName: example
  condition: always()
  timeoutInMinutes: 1
  retryCountOnTaskFailure: 10

Task only fail one only fails 1 time:

Thanks!