grass.script: Always use env for shutil.which

[wenzeslaus]

Currently, shutil.which is called on Windows to find the executable. However, it does not use the provided env (if any) which leads to the executable not being find if only the env parameter for run_command (and family) contains the runtime setup. This is not a problem on Linux (and probably unix/posix in general), because the Popen call eventually uses the system to find the executable and uses the provided env. However, on Windows, env is not used by the subprocess call. The shutil.which uses the os.environ process, so when the global environment is used, everything still works even on Windows.

This adds the PATH from env to the shutil.which call if it is there, so that it is used to find the exectuable. Additionally, this adds the shutil.which call for all platforms which is considered a best practice, avoiding Bandit's B607: Test for starting a process with a partial path.

https://bandit.readthedocs.io/en/latest/plugins/b607_start_process_with_partial_path.html

[echoix]

Do you have an idea on what to use to test it? I'll try it locally, but I don't have an idea in mind

[wenzeslaus]

On Windows, calling create_project before grass.script.setup.init presumably fails on main and in 8.4 with g.proj not found while on Linux it works.

[landam]

@wenzeslaus I run the script below from OSGeo4W shell:

import os
import sys
import subprocess

# Append GRASS to the python system path
sys.path.append(
    subprocess.check_output(["grass84.bat", "--config", "python_path"], text=True).strip()
)

import grass.script as gs

# Create a new project
project_path = os.path.join(os.environ["HOMEPATH"], "Documents", "grassdata", "pytest")
gs.create_project(path=project_path, epsg="3358")

# Initialize the GRASS session
with gs.setup.init(project_path) as session:
    # Run GRASS tools
    gs.run_command("g.region", flags="p")

It still fails with an error:

Traceback (most recent call last):
  File "c:\Users\martin\Downloads\grass_create_project.py", line 15, in <module>
    gs.create_project(path=project_path, epsg="3358")
  File "C:\OSGeo4W\apps\grass\grass84\etc\python\grass\script\core.py", line 1820, in create_project
    ps = pipe_command(
         ^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\grass\grass84\etc\python\grass\script\core.py", line 513, in pipe_command
    return start_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\grass\grass84\etc\python\grass\script\core.py", line 434, in start_command
    return Popen(args, **popts)
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\grass\grass84\etc\python\grass\script\core.py", line 56, in __init__
    raise OSError(_("Cannot find the executable {0}").format(args[0]))
OSError: Cannot find the executable g.proj

because the path doesn't contain in my case C:\OSGeo4W\apps\grass84\bin:

C:\OSGeo4W\apps\qt5\bin;C:\OSGeo4W\apps\Python312\Scripts;C:\OSGeo4W\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\WBem

[landam]

@wenzeslaus I tested it on GRASS 8.5 (a6b8391) with the same result.

[wenzeslaus]

Does the OSGeo shell set GISBASE? If yes, that would skip the runtime env setup, and thus fail later.

[wenzeslaus]

Is there a way you could transfer your local test to CI? ie it would fail on current main, but would be disabled, or edit this branch.

[echoix]

On Windows, calling create_project before grass.script.setup.init presumably fails on main and in 8.4 with g.proj not found while on Linux it works.

Is this related to something like this test (on main) that fails: https://github.com/OSGeo/grass/actions/runs/15363637275/job/43233794562#step:14:105
.\general\g.proj\testsuite\test_g_proj.py...

Running .\general\g.proj\testsuite\test_g_proj.py...
========================================================================
..F..

======================================================================

FAIL: test_proj_info_output_json (__main__.GProjTestCase.test_proj_info_output_json)

Test if g.proj returns consistent projection info in JSON format.

----------------------------------------------------------------------

Traceback (most recent call last):

  File "general\g.proj\testsuite\test_g_proj.py", line 94, in test_proj_info_output_json

    self.assert_keys_in_output(result)

  File "general\g.proj\testsuite\test_g_proj.py", line 37, in assert_keys_in_output

    self.assertIn(prefix + key, output)

AssertionError: 'proj' not found in {}



----------------------------------------------------------------------

Ran 5 tests in 0.767s

FAILED (failures=1)

========================================================================
FAILED .\general\g.proj\testsuite\test_g_proj.py (1 test failed)

[echoix]

Ok, here's what I got.

I tried with the following script, I named it d:\test2\pr5717.py:

import os
import sys
import subprocess

# Append GRASS to the python system path
# sys.path.append(
    # subprocess.check_output(["grass85.bat", "--config", "python_path"], text=True).strip()
# )

import grass.script as gs

# Create a new project
project_path = os.path.join(os.environ["HOMEPATH"], "Documents", "grassdata", "pytest")
gs.create_project(path=project_path, epsg="3358")

# Initialize the GRASS session
with gs.setup.init(project_path) as session:
    # Run GRASS tools
    gs.run_command("g.region", flags="p")

OSGeo4W Shell, without this PR applied:

run o-help for a list of available commands
C:\>python --version
Python 3.12.10

C:\>python d:\test2\pr5717.py
Traceback (most recent call last):
  File "d:\test2\pr5717.py", line 10, in <module>
    import grass.script as gs
ModuleNotFoundError: No module named 'grass'

C:\>echo "That was before changing anything"
"That was before changing anything"

Then, apply this PR's changes, then in a new OSGeo4W shell:

run o-help for a list of available commands
C:\>echo "With changes to the python/grass/script/core.py file"
"With changes to the python/grass/script/core.py file"

C:\>python d:\test2\pr5717.py
Traceback (most recent call last):
  File "d:\test2\pr5717.py", line 10, in <module>
    import grass.script as gs
ModuleNotFoundError: No module named 'grass'

Ok, so maybe setting the pythonpath was really needed? (Of course, as that shell makes sure to have a minimal PATH and some env vars set to some minimum, (but runs all of the .bat in : C:\OSGeo4W\etc\ini), and when starting the grass programs, adds more:

So, I then created the file d:\test2\pr5717_pythonpath.py:

import os
import sys
import subprocess

# Append GRASS to the python system path
sys.path.append(
    subprocess.check_output(["grass85.bat", "--config", "python_path"], text=True).strip()
)

import grass.script as gs

# Create a new project
project_path = os.path.join(os.environ["HOMEPATH"], "Documents", "grassdata", "pytest")
gs.create_project(path=project_path, epsg="3358")

# Initialize the GRASS session
with gs.setup.init(project_path) as session:
    # Run GRASS tools
    gs.run_command("g.region", flags="p")

In the same second osgeo4w shell, I ran:

C:\>python d:\test2\pr5717_pythonpath.py
projection: 99 (NAD83(HARN) / North Carolina)
zone:       0
datum:      nad83harn
ellipsoid:  grs80
north:      1
south:      0
west:       0
east:       1
nsres:      1
ewres:      1
rows:       1
cols:       1
cells:      1

C:\>python d:\test2\pr5717_pythonpath.py
ERREUR : Location <pytest> already exists. Operation canceled.

C:\>

Yay!

Let's make sure the new script also doesn't work in the first OSGeo4W shell, with the previous contents of grass/script/core:

C:\>echo "The following if after reverting to the previous code, but running a separate script setting pythonpath"
"The following if after reverting to the previous code, but running a separate script setting pythonpath"

C:\>python d:\test2\pr5717_pythonpath.py
Traceback (most recent call last):
  File "d:\test2\pr5717_pythonpath.py", line 14, in <module>
    gs.create_project(path=project_path, epsg="3358")
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 1841, in create_project
    fatal(
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 845, in fatal
    error(msg, env=env)
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 828, in error
    message(msg, flag="e", env=env)
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 747, in message
    run_command("g.message", flags=flag, message=msg, errors="ignore", env=env)
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 525, in run_command
    ps = start_command(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 480, in start_command
    return Popen(args, **popts)
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 88, in __init__
    raise OSError(_("Cannot find the executable {0}").format(args[0]))
OSError: Cannot find the executable g.message

C:\>

So, I'm not getting the same error as Martin, but g.message isn't even found for displaying something.

To make sure anything wasn't screwed up, let's open a 3rd OSGeo4W shell, and try witthout changing anything (without this PR applied) (note that I was commenting/uncommenting the changes at some point, so the line numbers don't always match):

run o-help for a list of available commands
C:\>echo "retry with the reverted code, that is, like distributed"
"retry with the reverted code, that is, like distributed"

C:\>python d:\test2\pr5717_pythonpath.py
Traceback (most recent call last):
  File "d:\test2\pr5717_pythonpath.py", line 14, in <module>
    gs.create_project(path=project_path, epsg="3358")
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 1841, in create_project
    fatal(
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 845, in fatal
    error(msg, env=env)
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 828, in error
    message(msg, flag="e", env=env)
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 747, in message
    run_command("g.message", flags=flag, message=msg, errors="ignore", env=env)
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 525, in run_command
    ps = start_command(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 480, in start_command
    return Popen(args, **popts)
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 88, in __init__
    raise OSError(_("Cannot find the executable {0}").format(args[0]))
OSError: Cannot find the executable g.message

C:\>

Let's try again, apply this PR, then open a 4th new OSGeo4W shell. I get the message that the location exists. Of course. Lets create a new file but renaming the location to create to "pytest2", it works.

run o-help for a list of available commands
C:\>echo "retry with this PR applied"
"retry with this PR applied"

C:\>python d:\test2\pr5717_pythonpath.py
ERREUR : Location <pytest> already exists. Operation canceled.

C:\>echo "renamed the location to create to pytest2"
"renamed the location to create to pytest2"

C:\>python d:\test2\pr5717_pythonpath2.py
projection: 99 (NAD83(HARN) / North Carolina)
zone:       0
datum:      nad83harn
ellipsoid:  grs80
north:      1
south:      0
west:       0
east:       1
nsres:      1
ewres:      1
rows:       1
cols:       1
cells:      1

C:\>

Since the g.message not found might have been because of the existing location, I created another script where the location is "pytest3" instead of "pytest2". I reverted the PR, openned a 5th OSGeo4W shell, and tried it:

run o-help for a list of available commands
C:\>echo "Reverted the PR, then try with a location that doesn't exist yet"
"Reverted the PR, then try with a location that doesn't exist yet"

C:\>python d:\test2\pr5717_pythonpath3.py
Traceback (most recent call last):
  File "d:\test2\pr5717_pythonpath3.py", line 14, in <module>
    gs.create_project(path=project_path, epsg="3358")
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 1846, in create_project
    ps = pipe_command(
         ^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 543, in pipe_command
    return start_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 464, in start_command
    return Popen(args, **popts)
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\grass\grass85\etc\python\grass\script\core.py", line 72, in __init__
    raise OSError(_("Cannot find the executable {0}").format(args[0]))
OSError: Cannot find the executable g.proj

C:\>

It gives the g.proj error that Martin had.

So, following this, I think this PR is definitely an improvement over what we have. Let us rerun an updated CI run, and since #5800 is not merged yet, we need to make sure at least 325/334 test files pass, to know if any other tests regress.

[echoix]

Considering my testing today, and the fact that the fresh CI run has 326/335 successful files, so 97.31%, it is fine to have this merged.

It didn't make more test files pass, but the project is better with this PR than without.

[echoix]

I had run CI of windows 2022 of #5562 with this PR, and was successful too.

[wenzeslaus]

This and four other related PRs are now merged (#5849, #5854, #5875, and #5920) including #5854 which has a little more robust check for existing environment (checking PATH too). Can you please try again? (@landam and/or @echoix)

Also, please see not only what GISBASE has in it, but also check PATH before calling create_project:

print(os.environ.get("GISBASE"))
print(os.environ.get("PATH"))