Reinstall module deps when they're missing from the environment

[liquidsec]

Problem

BBOT pip-installs module deps into the active environment and records success in ~/.bbot/cache/depsinstaller/setup_status.json, keyed by a hash of the module's deps + venv path + bbot home + hostname + version. That hash never changes when the contents of the venv change.

So when the venv gets rebuilt out from under BBOT (uv sync prunes anything not in the lockfile, which includes every module deps_pip), the packages are gone but the cache still says "installed". BBOT skips the reinstall, reports the module as set up, and the module then fails to import. baddns is a frequent casualty, but this hits every module with pip deps (wafw00f, markdown, neo4j, pymongo, sqlmodel, ...).

The only known workaround was wiping ~/.bbot by hand.

Reproducer against dev:

$ uv pip uninstall baddns
$ python -c "... await installer.install('baddns')"
before: baddns MISSING
succeeded: ['baddns'] failed: []
after: baddns MISSING      # <-- reported success, installed nothing

Fix

Before honoring a cache hit, check that the module's pip requirements are actually satisfied in the current environment (importlib.metadata + packaging.requirements). If a package is missing or at a non-satisfying version, the cache entry is treated as a miss and deps are reinstalled.

This mirrors how install_core_deps already works: it checks whether each binary is present rather than trusting a flag.

Requirements that can't be parsed (VCS URLs) are assumed satisfied, and requirements whose environment markers don't apply are skipped, so neither causes a reinstall loop.

Same reproducer after the fix:

[INFO] Installing module dependencies. Please be patient, this may take a while.
[INFO] Installing the following pip packages: baddns~=2.4.0
[INFO] Successfully installed baddns-2.4.1
before: baddns MISSING
succeeded: ['baddns'] failed: []
after: 2.4.1

Also in here

• packaging promoted to a direct dependency. It was already present transitively via ansible-core/ansible-runner, but core code shouldn't rely on a transitive dep. Lockfile diff is 2 lines, no version churn.
• Dead else branch removed from install()'s dependency loop. The preceding branch always continues, so the else was unreachable structure; collapsing it flattens the nested if/else into elif. This is why the installer diff looks larger than the fix itself.
• pip_install now honors custom constraints. The condition was inverted (if constraints is not None: constraints = get_python_constraints()), which replaced a caller's custom constraints with bbot's own instead of using bbot's as the fallback. No module sets deps_pip_constraints today, so this changes no current behavior, but the code contradicted its own comment.

Tests

test_depsinstaller_stale_pip_cache fails on dev (assert [] == [['bbot-nonexistent-package']]) and passes here. It also asserts the cache is still honored for deps that are genuinely installed, so the fix can't degrade into reinstalling on every scan. test_depsinstaller_pip_constraints likewise fails against the old inverted condition. Plus unit coverage of the requirement check itself (missing, version mismatch, non-applicable marker, unparseable).

Note on sudo prompts

A module with both deps_apt and deps_pip (e.g. sslcert) now re-runs its whole install when only the pip half went missing, which can prompt for a sudo password. That's identical to post-rm -rf ~/.bbot behavior, so it's a return to correct behavior rather than a regression, but users who never saw a prompt after a uv sync may now see one.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs stale-deps-cache (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.23ms	4.23ms	+0.0% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	17.52ms	17.71ms	+1.1% ⚪	✅
Large Closest Match Lookup	364.88ms	369.22ms	+1.2% ⚪	✅
Realistic Closest Match Workload	191.10ms	196.02ms	+2.6% ⚪	✅
Event Memory Medium Scan	1402 B/event	1402 B/event	-0.0% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	406.99ms	416.06ms	+2.2% ⚪	✅
Event Validation Full Scan Startup Large Batch	548.60ms	551.50ms	+0.5% ⚪	✅
Make Event Autodetection Small	25.94ms	26.07ms	+0.5% ⚪	✅
Make Event Autodetection Large	264.35ms	264.90ms	+0.2% ⚪	✅
Make Event Explicit Types	11.67ms	11.64ms	-0.2% ⚪	✅
Excavate Single Thread Small	4.470s	4.144s	-7.3% ⚪	✅
Excavate Single Thread Large	10.240s	10.045s	-1.9% ⚪	✅
Excavate Parallel Tasks Small	4.669s	4.334s	-7.2% ⚪	✅
Excavate Parallel Tasks Large	6.748s	6.450s	-4.4% ⚪	✅
Intercept Throughput Small	887.50ms	925.52ms	+4.3% ⚪	✅
Intercept Throughput Medium	874.52ms	876.13ms	+0.2% ⚪	✅
Dns Throughput Quiet	2.782s	2.842s	+2.2% ⚪	✅
Dns Throughput Loaded	1.831s	1.801s	-1.7% ⚪	✅
Dns Throughput Inherited	2.549s	2.499s	-2.0% ⚪	✅
Is Ip Performance	2.30ms	2.23ms	-2.9% ⚪	✅
Make Ip Type Performance	241.28µs	235.57µs	-2.4% ⚪	✅
Mixed Ip Operations	2.40ms	2.33ms	-3.0% ⚪	✅
Memory Use Web Crawl	1.0 MB	1.0 MB	-1.0% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	10.1 MB	15.1 MB	+49.6% 🔴🔴🔴	⚠️
Scan Throughput 100	2.839s	2.815s	-0.8% ⚪	✅
Scan Throughput 1000	21.328s	21.778s	+2.1% ⚪	✅
Typical Queue Shuffle	5.39µs	5.52µs	+2.4% ⚪	✅
Priority Queue Shuffle	25.92µs	26.01µs	+0.3% ⚪	✅

🎯 Performance Summary

! 1 regression ⚠️
  30 unchanged ✅

🔍 Significant Changes (>10%)

• Memory Use Parallel Chains: 49.6% 🐌 more memory

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

❌ Patch coverage is 96.63866% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (fab03be) to head (bd6a281).
⚠️ Report is 26 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/core/helpers/depsinstaller/installer.py	91%	4 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3338   +/-   ##
=====================================
+ Coverage     90%     90%   +1%     
=====================================
  Files        450     450           
  Lines      46308   46407   +99     
=====================================
+ Hits       41565   41660   +95     
- Misses      4743    4747    +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[singlerider]

Verified both fixes.

Constraints inversion is real and dates to cec3cb4 (Oct 2024). Unnoticed because no module sets deps_pip_constraints, so only the falsy path was reachable. Reverting that one line fails the new test, so it gates properly.

Stale cache check: ran it against all 18 real module pip deps, zero false reinstalls. Edge cases correct. ~4ms per module at startup. The install() de-indent is behavior preserving.

Two non-blocking notes: pip-only, so it is a no-op for the 12 modules with only non-pip deps (dnsbrute, portscan, jadx, trufflehog, etc). And extras are parsed but unused, so pkg[extra] reads as satisfied. No module uses extras today.