Description
What happened?
I am using the new sparseCheckoutDirectories under my checkout tasks. When used in conjunction with fetchFilter: blob:none (or fetchFilter: tree:0), helps to exclude and trim back all unnecessary files and keep the checkout download size to a minimum. However, I've noticed that if I amend sparseCheckoutDirectories to include an additional directory AFTER the pipeline has already been run on the target environment, then we will receive the following error:
git sparse-checkout set MyFolder1 MyFolder2
fatal: Cannot prompt because terminal prompts have been disabled.
fatal: could not read Password for 'https://****@dev.azure.com'/: terminal prompts disabled
fatal: could not fetch **** from promisor remote
##[error]Git sparse checkout failed with exit code: 128
No clue why it's prompting for credentials in the first place...
Example checkout task (first run = success):
- checkout: az-pipeline-scripts
fetchFilter: tree:0
sparseCheckoutDirectories: MyFolder1
path: repo-folder/Example modified checkout task (second run = fails):
- checkout: az-pipeline-scripts
fetchFilter: tree:0
sparseCheckoutDirectories: MyFolder1 MyFolder2
path: repo-folder/Since our deployment agents are self-hosted and persist the working directory in-between runs, I have to manually go into the server and wipe the repo folder to get this to work. I have tried various combinations of settings, included clean: true, however the error will ALWAYS persists, unless I either a) delete the target repo folder prior to checkout task, or b) reset the task back to it's original settings (which of course is moving backwards from the goal).
From my testing, if I exclude the fetchFilter from the start, then subsequent changes will work without issue. However the goal here is to minimize the size of the checked out repo and hidden pack files. In my particular scenario, one folder has several 100mb blobs which also tend to increase the hidden pack folder, even when using sparse-checkout. So to optimize, we must use the combination of a filter and sparse-checkout.
Honestly, the main issue here is that the agent work folder for this pipeline remains in a broken state after the first run. I would have expected that clean option to have helped, however I don't see it doing anything. After that first run executes, it seems impossible to even load the full repo (again w/o deleting repo folder). For example , this still fails after the first run (notice how filter and sparse-checkout are excluded):
- checkout: az-pipeline-scripts
clean: true
path: repo-folder/Issue #5113 claims something somewhat similar, however w/o more logs/details, I suspect the core issue is different. The fix suggested in that issue (adjusting order of fetch/sparse commands) has already been shipped with my agent's version.
Versions
OS Version: Microsoft Windows Server 2019 Version 1809 (OS Build 10.0.17763.5458)
Current agent version: '4.252.0'
Checking job knob settings.
Knob: DockerActionRetries = true Source: $(VSTSAGENT_DOCKER_ACTION_RETRIES)
Knob: UseGitLongPaths = true Source: $(USE_GIT_LONG_PATHS)
Knob: EnableIssueSourceValidation = true Source: $(ENABLE_ISSUE_SOURCE_VALIDATION)
Knob: AgentEnablePipelineArtifactLargeChunkSize = true Source: $(AGENT_ENABLE_PIPELINEARTIFACT_LARGE_CHUNK_SIZE)
Knob: ContinueAfterCancelProcessTreeKillAttempt = true Source: $(VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT)
Knob: ProcessHandlerSecureArguments = false Source: $(AZP_75787_ENABLE_NEW_LOGIC)
Knob: ProcessHandlerSecureArguments = false Source: $(AZP_75787_ENABLE_NEW_LOGIC_LOG)
Knob: ProcessHandlerTelemetry = true Source: $(AZP_75787_ENABLE_COLLECT)
Knob: UseNewNodeHandlerTelemetry = True Source: $(DistributedTask.Agent.USENEWNODEHANDLERTELEMETRY)
Knob: ProcessHandlerEnableNewLogic = true Source: $(AZP_75787_ENABLE_NEW_PH_LOGIC)
Knob: EnableResourceMonitorDebugOutput = true Source: $(AZP_ENABLE_RESOURCE_MONITOR_DEBUG_OUTPUT)
Knob: EnableResourceUtilizationWarnings = true Source: $(AZP_ENABLE_RESOURCE_UTILIZATION_WARNINGS)
Knob: IgnoreVSTSTaskLib = true Source: $(AZP_AGENT_IGNORE_VSTSTASKLIB)
Knob: FailJobWhenAgentDies = true Source: $(FAIL_JOB_WHEN_AGENT_DIES)
Knob: CheckForTaskDeprecation = true Source: $(AZP_AGENT_CHECK_FOR_TASK_DEPRECATION)
Knob: CheckIfTaskNodeRunnerIsDeprecated246 = False Source: $(DistributedTask.Agent.CheckIfTaskNodeRunnerIsDeprecated246)
Knob: UseNode20ToStartContainer = True Source: $(DistributedTask.Agent.UseNode20ToStartContainer)
Knob: LogTaskNameInUserAgent = true Source: $(AZP_AGENT_LOG_TASKNAME_IN_USERAGENT)
Knob: UseFetchFilterInCheckoutTask = true Source: $(AGENT_USE_FETCH_FILTER_IN_CHECKOUT_TASK)
Knob: Rosetta2Warning = true Source: $(ROSETTA2_WARNING)
Knob: AddForceCredentialsToGitCheckout = True Source: $(DistributedTask.Agent.AddForceCredentialsToGitCheckout)
Knob: UseSparseCheckoutInCheckoutTask = true Source: $(AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK)
Finished checking job knob settings.
AGENT_VERSION: 4.252.0
AGENT_ISSELFHOSTED: 1
AGENT_OS: Windows_NT
AGENT_OSARCHITECTURE: X64
AGENT_USE_FETCH_FILTER_IN_CHECKOUT_TASK: true
AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK: true
Environment type (Please select at least one enviroment where you face this issue)
- Self-Hosted
- Microsoft Hosted
- VMSS Pool
- Container
Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
Microsoft Windows Server 2019 Version 1809 (OS Build 10.0.17763.5458)
Version controll system
Azure Git
Relevant log output
Starting: Checkout AzurePipelinesScripts@main to repo-folder/
==============================================================================
Task : Get sources
Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version : 1.0.0
Author : Microsoft
Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Syncing repository: AzurePipelinesScripts (git)
Prepending Path environment variable with directory containing 'git.exe'.
git version
git version 2.47.0.windows.2
git lfs version
git-lfs/3.4.0 (GitHub; windows amd64; go 1.20.6; git d06d6e9e)
git config --get remote.origin.url
git clean -ffdx
git reset --hard HEAD
HEAD is now at 71e6b15 Dummy push for Pipeline
git sparse-checkout init --cone
git sparse-checkout set MyFolder1 MyFolder2
fatal: Cannot prompt because terminal prompts have been disabled.
fatal: could not read Password for 'https://****@dev.azure.com': terminal prompts disabled
fatal: could not fetch **** from promisor remote
##[error]Git sparse checkout disable failed with exit code: 128
Finishing: Checkout AzurePipelinesScripts@main to repo-folder/