Trouble installing moose - all tests fail

[eshemon]

Our cluster recently went through an upgrade (I'm mentioning this in case something wonky happened that might be related to this). Anyways, I'm building MOOSE from scratch using https://mooseframework.inl.gov/getting_started/installation/conda.html

Everything works perfectly/as expected until I execute "./run_tests -j 4" at which point all tests fail. The error given for basically all tests is FAILED (Application not found).

I followed the troubleshooting guide https://mooseframework.inl.gov/help/troubleshooting.html#failingtests. However, the "Application not found" issue does not seem to be the problem? Make works, and a valid moose_test-opt is present, which can be executed and run a sample problem. I performed all the debugging in the Build section (hello world, etc). Everything seems fine?

Need help understanding why all tests are failing and any suggestions how to fix it. Screen output is attached from the failed tests and debugging attempts...
moose_build_errors.txt

[permcody]

This is odd. I haven't seen this happen before, but I know that the logic to build or discover what the executable name should be can be fairly complex. It relies on various information from the platform, the directory structure and can be influenced by the "testroot" file too. There haven't been any changed merged to the TestHarness or testers in the last 2-3 months.

What I would suggest is that we attempt to printout what the TestHarness "thinks" the name of the binary should be and go from there. It appears that much of the logic for figuring out the app name occurs here. Adding a simple print statement after this block should do it:

moose/python/TestHarness/TestHarness.py

Lines 881 to 902 in 7aeadfc

	## Save executable-under-test name to self.executable
	exec_suffix = 'Windows' if platform.system() == 'Windows' else ''
	self.executable = app_name + '-' + self.options.method + exec_suffix
	# if the executable has a slash - assume it is a file path
	if '/' in app_name:
	self.executable = os.path.abspath(self.executable)
	# look for executable in PATH - if not there, check other places.
	elif shutil.which(self.executable) is None:
	if os.path.exists(os.getcwd() + '/' + self.executable):
	# it's in the current working directory
	self.executable = os.getcwd() + '/' + self.executable
	elif os.path.exists(os.path.join(self._rootdir, self.executable)):
	# it's in the testroot file's directory
	self.executable = self.executable
	# we may be hopping around between multiple (module)
	# subdirectories of tests - so the executable needs to be an
	# absolute path.
	self.executable = os.path.abspath(os.path.join(self._rootdir, self.executable))
	else:
	# it's (hopefully) in an installed location
	mydir = os.path.dirname(os.path.realpath(__file__))
	self.executable = os.path.join(mydir, '../../../..', 'bin', self.executable)

@aeslaughter or @milljm, any thoughts?

[eshemon]

OK - I printed self.executable after line 902:

START DEBUG PRINT: TestHarness.py printing self.executable:
moose_test-opt
END DEBUG PRINT

Is there supposed to be a path in front of that?

[eshemon]

Cody suggested running ./run_tests -j4 -v --dry-run to see if the executable path was correct. The running command includes an executable name without a full path - possibly the issue?

dirackernels/material_point_source.material_point_source_test: Working Directory: /beegfs1/home/eshemon/projects/moose/test/tests/dirackernels/material_point_sour                                                                                                                                     ce
dirackernels/material_point_source.material_point_source_test: Running command: moose_test-opt -i material_point_source.i --error --error-unused --error-override                                                                                                                                      --no-gdb-backtrace

[permcody]

Well your PYTHON_PATH isn't set, that was another idea we had.

I'd like to continue to troubleshoot, but as a quick workaround you might just (temporarily) export the path to the binary you are trying to run. It's a hack but it should make all your tests pass.

export PATH=$PATH:/beegfs1/home/eshemon/projects/moose/test/

[eshemon]

Thanks Cody, exporting the path definitely works as a quick workaround. Tests passing now.

[eshemon]

Any update on why this issue was happening and a more permanent fix? I'm happy to debug online via remote call if that's easier.

[milljm]

Your PATH (was) is set to:

PATH=/home/eshemon/miniconda3/envs/moose/bin:/home/eshemon/miniconda3/condabin:.:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/eshemon/bin

Can you remove the path to current working directory? :.: Basically make your path's value:

export PATH=/home/eshemon/miniconda3/envs/moose/bin:/home/eshemon/miniconda3/condabin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/eshemon/bin

Sometimes there exists aliases, symlinks, or some 'thing' in a current working directory which may end up finding stuff that we don't want. I realize people add a path like this so they don't have to invoke binaries with a prefixed ./, but the trouble this sometimes causes is not worth it. Not sure if this is your issue, but it's worth a shot.

[eshemon]

I forgot to update this conversation, but I wanted to let you know that your suggestion worked. I removed the path to the current working directory and it fixed everything. Thanks for the help.