Pants 2.24 - IntrinsicError: Not a directory (os error 20)

[chris-smith-zocdoc]

Describe the bug
If a symlink to a file (not a directory) is on the PATH for finding system binaries, pants errors out with IntrinsicError: Not a directory (os error 20)

One of my coworkers encountered this error while using the docker backend I think. The easiest reproduction I could come up is using the system_binary target though

touch my_file
ln -s my_file my_symlink

system_binary(
    name="tool",
    binary_name="ls",
    extra_search_paths=[
        "/var/abs/path/my_symlink",
    ],
)

pants run :tool 

Pants version
2.24.1

OS
MacOS

Additional info

Offending code is here

Similar, but not the same, as #21954

Traceback (most recent call last):
  File "<snip>/pants/core/goals/run.py", line 242, in run
    request = await (
  File "<snip>/pants/engine/internals/selectors.py", line 124, in __await__
    result = yield self
  File "<snip>/pants/backend/adhoc/run_system_binary.py", line 168, in create_system_binary_run_request
    path = await _find_binary(
  File "<snip>/pants/backend/adhoc/run_system_binary.py", line 72, in _find_binary
    binaries = await Get(
  File "<snip>/pants/engine/internals/selectors.py", line 124, in __await__
    result = yield self
  File "<snip>/pants/core/util_rules/system_binaries.py", line 776, in find_binary
    found_paths = await _find_candidate_paths_via_path_metadata_lookups(request)
  File "<snip>/pants/core/util_rules/system_binaries.py", line 673, in _find_candidate_paths_via_path_metadata_lookups
    file_metadata_results = await MultiGet(
  File "<snip>/pants/engine/internals/selectors.py", line 376, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "<snip>/pants/engine/internals/selectors.py", line 174, in __await__
    result = yield self.gets
  File "<snip>/pants/engine/intrinsics.py", line 204, in path_metadata_request
    return await native_engine.path_metadata_request(request)
native_engine.IntrinsicError: Not a directory (os error 20)

In pants 2.23.2 my example succeeds

[tgolsson]

I'd imagine there's a lot of entries you can put in an environment variable that'll make Pants (and other programs) barf. Putting a file entry in $PATH is out of spec (emphasis mine):

2. The shell searches each entry in PATH in turn for the command. The value of the PATH variable should be a series of entries separated by colons. Each entry consists of a directory name. The current directory may be indicated implicitly by an empty directory name, or explicitly by a single period.

A similar issue can occur in WSL where your $PATH can contain entries pointing to the Windows directory tree, and some of those may have names that match -- but they're not actually valid Linux binaries. In my world that isn't a bug in Pants, it's a misconfiguration of my system.

(Which isn't to say we shouldn't fix this regression, but doing anything except ignoring/logging is superfluous.)