Preflight Checklist
What's Wrong?
While debugging a PDF-parsing script (subprocess.run(["pdftotext", ...]), invoked via Python), the agent iteratively diagnosed and "fixed" several apparent logic bugs across multiple commits — based on testing done entirely within its own bash tool environment.
The user later reported that running the same script via double-clicking a .bat file (which runs under cmd.exe) produced different, wrong results than when the agent ran it. It turned out the machine had two different pdftotext.exe binaries on PATH (one bundled with Git for Windows, one installed separately via a package manager) — the agent's bash environment resolved one of them by PATH order, while cmd.exe resolved the other. The two binaries render ambiguous PDF layout differently, so the same parsing code produced different output depending on which binary happened to run — which looked exactly like a logic bug but wasn't one.
The agent spent significant effort (many tool calls, several commits) diagnosing and fixing what it believed were parsing bugs, including reporting a large fictitious financial discrepancy, before the user pointed out the environment mismatch. Once caught, the fix was simple (pin the external tool to an explicit path) and most of the "bug fixes" turned out to be unnecessary — the original logic was correct all along.
What Should Happen?
when an agent is debugging code that shells out to an external command-line tool by bare name, it should proactively check whether more than one binary of that name exists on PATH, and whether the same one resolves in the environment where the user will actually run the code (vs. the agent's own tool-execution environment), before concluding the invoking code's logic is broken.
Error Messages/Logs
No exception was ever thrown — that's the core of the problem. Concrete evidence of the divergence:
Running pdftotext -layout on the identical PDF page with each binary produced structurally different text for the same table row (e.g. a service-description label and its amount landing on different output lines, or a value shifting to an adjacent row), causing downstream regex-based parsing to extract different, silently-wrong numbers depending solely on which binary happened to run.
Suggested improvement: when an agent is debugging code that shells out to an external command-line tool by bare name, it should proactively check whether more than one binary of that name exists on PATH, and whether the same one resolves in the environment where the user will actually run the code (vs. the agent's own tool-execution environment), before concluding the invoking code's logic is broken.
Steps to Reproduce
On a Windows machine, have two differently-behaving executables with the same name on PATH (e.g. two pdftotext.exe builds from different projects — Xpdf vs. Poppler — that render the same input differently for edge cases).
Ask the agent (via its Bash tool) to debug/fix a script that shells out to that tool by bare name (subprocess.run(["sometool", ...])) and parses its text output with regex.
The agent's own Bash tool environment resolves a different one of the two binaries (via PATH order) than the environment the user actually runs the script in (e.g. cmd.exe via a double-clicked .bat).
The agent, testing exclusively inside its own tool environment, diagnoses "bugs" in the parsing logic and makes several rounds of code changes to fix them — without checking whether more than one binary of that name exists on PATH, or whether its own resolution matches the user's.
The user runs the "fixed" code themselves and reports it still produces the original wrong output.
Only after repeated back-and-forth does the agent discover the mismatch, by explicitly running where / Get-Command -All.
Claude Model
Sonnet (default)
Is this a regression?
I don't know
Last Working Version
No response
Claude Code Version
2.1.218
Platform
Anthropic API
Operating System
Windows
Terminal/Shell
Windows Terminal
Additional Information
No response
Preflight Checklist
What's Wrong?
While debugging a PDF-parsing script (subprocess.run(["pdftotext", ...]), invoked via Python), the agent iteratively diagnosed and "fixed" several apparent logic bugs across multiple commits — based on testing done entirely within its own bash tool environment.
The user later reported that running the same script via double-clicking a .bat file (which runs under cmd.exe) produced different, wrong results than when the agent ran it. It turned out the machine had two different pdftotext.exe binaries on PATH (one bundled with Git for Windows, one installed separately via a package manager) — the agent's bash environment resolved one of them by PATH order, while cmd.exe resolved the other. The two binaries render ambiguous PDF layout differently, so the same parsing code produced different output depending on which binary happened to run — which looked exactly like a logic bug but wasn't one.
The agent spent significant effort (many tool calls, several commits) diagnosing and fixing what it believed were parsing bugs, including reporting a large fictitious financial discrepancy, before the user pointed out the environment mismatch. Once caught, the fix was simple (pin the external tool to an explicit path) and most of the "bug fixes" turned out to be unnecessary — the original logic was correct all along.
What Should Happen?
when an agent is debugging code that shells out to an external command-line tool by bare name, it should proactively check whether more than one binary of that name exists on PATH, and whether the same one resolves in the environment where the user will actually run the code (vs. the agent's own tool-execution environment), before concluding the invoking code's logic is broken.
Error Messages/Logs
Steps to Reproduce
On a Windows machine, have two differently-behaving executables with the same name on PATH (e.g. two pdftotext.exe builds from different projects — Xpdf vs. Poppler — that render the same input differently for edge cases).
Ask the agent (via its Bash tool) to debug/fix a script that shells out to that tool by bare name (subprocess.run(["sometool", ...])) and parses its text output with regex.
The agent's own Bash tool environment resolves a different one of the two binaries (via PATH order) than the environment the user actually runs the script in (e.g. cmd.exe via a double-clicked .bat).
The agent, testing exclusively inside its own tool environment, diagnoses "bugs" in the parsing logic and makes several rounds of code changes to fix them — without checking whether more than one binary of that name exists on PATH, or whether its own resolution matches the user's.
The user runs the "fixed" code themselves and reports it still produces the original wrong output.
Only after repeated back-and-forth does the agent discover the mismatch, by explicitly running where / Get-Command -All.
Claude Model
Sonnet (default)
Is this a regression?
I don't know
Last Working Version
No response
Claude Code Version
2.1.218
Platform
Anthropic API
Operating System
Windows
Terminal/Shell
Windows Terminal
Additional Information
No response