Builds hanging forever following a symlink

[pdohertybcov]

After installing the browserstack 2.0.0 plugin our builds hang indefinitely. (A build that used to take 2 minutes is still running after 20 hours)

We think it's related to this code:
https://github.com/browserstack/teamcity-plugin/blob/master/browserstack-teamcity-agent/src/main/java/com/browserstack/automate/ci/teamcity/BrowserStackLocalAgent.java#L109-L111

And may be related to this issue: #16

The builds that triggered this have a symlink that points to the parent directory, which seems to result in a state of infinite recursion when the plugin starts inspecting the directory. The build steps will finish, and then the build just hangs indefinitely with the Java process on the agent running at 100% CPU. Using strace on the java process shows it running stat on an ever expanding path like foo/bar/foo/bar/foo/bar/foo/bar...

A simple reproduction is a build like so:

#!/usr/bin/env bash
set -euo pipefail
set -x
ln -s . ./here
ln -s . ./here2
ls -al ./
ls -al here/
ls -al here2/
echo "Success!  But the agent will hang after this script successfully finishes."

Can this be fixed? It's completely blocking us from being able to use the plugin.

Thanks!

Hey @pdohertybcov,
Could you share your use case with us?

[pdohertybcov]

We have a git repository that has a git submodule located at .foo

The git submodule includes a symlink at .foo/bar that links to ..

The git submodule links to itself. This was needed for reasons I don't know all the details of, but I know it's needed.

Running builds using any of the repositories that include the git submodule cause the build to hang. We had to disable the browserstack plugin in order to get the builds running again.

Thanks.

[cspotcode]

For what it's worth, we don't use the browserstack plugin in the affected builds. We could pretty easily add a parameter to our project to disable the plugin, if such a parameter existed.

[abhi291096]

Hello, this seems like an issue with TeamCity in general. I would recommend you to raise an issue in the link: https://www.jetbrains.com/help/teamcity/reporting-issues.html

[cspotcode]

I have a support ticket open with JetBrains. They're going to investigate FileUtil internally.

In the meantime, can we get a parameter that lets us disable the FileUtil search for certain builds? I see you already set an isEnabled field based on the presence of a build feature. What about another field that checks for a parameter instead of a build feature, and when present, disables the search for artifacts. It will still default to on, which preserves backwards compatibility, but the parameter will let us disable it in the affected projects.

[cspotcode]

TeamCity support opened a ticket about the bug but also mentioned that FileUtil is deprecated. I didn't confirm this myself, but I'm passing it along.

I think implementing a parameter that allows disabling the plugin in some projects is still a good idea.

EDIT: they said FileUtil.collectMatchedFiles is deprecated.

[RohanImmanuel]

@cspotcode you can disable the plugin by removing it from the "Build Features" in the "Build Configuration Settings"

[cspotcode]

This part of the plugin always executes on all builds, with or without the build feature. You can see this in the code, since it is not gated behind the isEnabled field.

…

On Thu, Jul 16, 2020, 10:20 AM Rohan Immanuel ***@***.***> wrote: @cspotcode <https://github.com/cspotcode> you can disable the plugin by removing it from the "Build Features" in the "Build Configuration Settings" — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OFOLQ7FVD4424GSYODR34EDHANCNFSM4OZ2NPWA> .

[RohanImmanuel]

The current implementation of this plugin attempts to generate reports by default, the option to skip this may be added as a feature in the near future. But till this is implemented, you may disable the plugin for this particular project by removing it from the Build Features (Not Build Steps). This will cause no part of the plugin code to be executed.

You will be required to set the environment variables BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY in your build steps before initializing your tests, documentation for the same is linked below:
https://confluence.jetbrains.com/display/TCD65/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-AddingorChangingaBuildParameterfromaBuildStep

Addadtionly, if you are using the plugin to initialize Local, you will have to eighter use language binding to do the same or use the below shell script in the Build Steps.
Start: ./path-to-lcoal/BrowserStackLocal --key ACCESS_KEY -d start
Stop: ./path-to-lcoal/BrowserStackLocal --key ACCESS_KEY -d stop

[cspotcode]

@RohanImmanuel that is incorrect; parts of the plugin execute even when the build does not have any "build features" attached. JetBrains support confirmed this. I spoke with Denis Lapuente.

You can confirm this by looking at your plugin's source code. Let me know if you want me to point out the relevant methods.

See also: #21 The error happens because the plugin is still executing on builds which do not have the build feature.

[RohanImmanuel]

@cspotcode I can assure you that the plugin only executes when its added as a build feature.

You can find Build Features in Build Configuration:

As for the fix #21, the error was caused as the BrowseStackLocal failed to initialized as reported here #20

[cspotcode]

@RohanImmanuel unfortunately, that is incorrect. When JetBrains reproduced this problem, they did it without the build feature.

I can see in the code that some parts of the plugin do not execute without the build feature, but the part I'm talking about runs regardless. It is this line of code:

teamcity-plugin/browserstack-teamcity-agent/src/main/java/com/browserstack/automate/ci/teamcity/BrowserStackLocalAgent.java

Lines 106 to 111 in e8631cd

	// find and collect report files that match our Ant pattern
	// if found, artifacts are published to the artifact directory by appending "=> <path>"
	List<File> reportFiles = new ArrayList<File>();
	FileUtil.collectMatchedFiles(build.getCheckoutDirectory(),
	Pattern.compile(FileUtil.convertAntToRegexp(REPORT_FILE_PATTERN)),
	reportFiles);

[RohanImmanuel]

@cspotcode to debug this faster can you reach out to support@browserstack.com, with a repository to help replicate the error.

[cspotcode]

We've removed the plugin from our TeamCity instance so this is not a pressing issue for us at the moment. Here is the reproduction that we sent to JetBrains.

Add the plugin to the server. Create a build. Do not add the build feature to the build. Add this script to the build.

#!/usr/bin/env bash
set -euo pipefail
set -x
ln -s . ./here
ln -s . ./here2
ls -al ./
ls -al here/
ls -al here2/
echo "Success! But the agent will hang after this script successfully finishes."

Run the build. The script will finish successfully, then the agent will hang.