Add Python 3 host environment execution support - #170
Conversation
Allows users to run scripts using the host's Python 3 environment, enabling the use of external pip packages. Adds a UI dropdown to select between Jython and Python 3. Retains full API parity with the legacy ScriptEnvironment.py.
|
Wow that's quite the PR. I love the concept, but I'm concerned about the potential tech debt of having two implementations of RequestEngine. Have you been using this heavily yourself? |
|
Hey @albinowax, thanks for taking the time to look at this! Just to clarify the tech debt concern —
Pre-built jar to try it out: https://github.com/vichhka-git/turbo-intruder/releases/tag/python3-v1.0 The main motivation is unlocking |
|
This is the section I was referring to - it looks like it'll break if there are any API changes. I'm not sure what the best fix is though: Other than that the code looks fairly clean, the main things to change are:
|
Replace custom SimpleJsonParser and jsonEncode() in Python3Runner with com.google.gson:gson:2.11.0 — a standard, widely-used JVM JSON library with no Burp API coupling.
- Restore show-eol and visible-whitespace to read from Utilities.globalSettings.getBoolean() instead of hardcoded false - Load enginePreference and python3Path via callbacks.loadExtensionSetting and save via callbacks.saveExtensionSetting (correct persistence layer for extension-panel UI controls) - Guard BurpRequestEngine BURP2 batch path against null HTTP responses: resp.response() can be null during race/gate attacks; calling toByteArray() on null caused a silent NPE that swallowed every response, resulting in 0 requests processed
There was a problem hiding this comment.
Pull request overview
This PR adds an alternate script execution mode that runs Turbo Intruder attack scripts in a host Python 3 subprocess (JSON-RPC), while keeping the existing Jython engine as the default.
Changes:
- Adds UI controls to select the scripting engine (Jython vs Python 3) and configure a Python 3 executable path.
- Introduces a
Python3RunnerJSON-RPC bridge and bundles aturbo_intruder.pyPython-side API stub. - Updates request engine handling for null Montoya responses and adds Gson + Kotlin JVM target config in Gradle.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
src/fast-http.kt |
Adds engine selection + Python path UI, and starts Python 3 runner instead of Jython when selected. |
src/Python3Runner.kt |
Implements the Python 3 subprocess JSON-RPC transport and bridges queue/response events into existing engines/UI. |
src/BurpRequestEngine.kt |
Adds null-safety around response handling when Burp/Montoya returns no response. |
resources/turbo_intruder.py |
Provides the Python 3 API stub (decorators, RequestEngine wrapper, response fetching, table.add). |
build.gradle |
Adds Gson dependency and Kotlin jvmTarget=21 configuration. |
AGENTS.md |
Adds a project knowledge base / contributor guidance doc. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| 1, 4 -> BurpRequestEngine(ep, connections, 2048, retries, idle, callback, null, false) | ||
| 3 -> HTTP2RequestEngine(ep, connections, 2048, rpc, retries, idle, callback, null) | ||
| else -> ThreadedRequestEngine( | ||
| ep, connections, 2048, 1, rpc, retries, idle, | ||
| callback, 20, null, 8192, false |
There was a problem hiding this comment.
handleCreateEngine reads fixContentLength into fixCL but never applies it (and the local is unused). If Python 3 scripts pass fixContentLength=False, it will be silently ignored. Either remove this param or plumb it through to engine/queue behavior consistently.
| 1, 4 -> BurpRequestEngine(ep, connections, 2048, retries, idle, callback, null, false) | |
| 3 -> HTTP2RequestEngine(ep, connections, 2048, rpc, retries, idle, callback, null) | |
| else -> ThreadedRequestEngine( | |
| ep, connections, 2048, 1, rpc, retries, idle, | |
| callback, 20, null, 8192, false | |
| 1, 4 -> BurpRequestEngine(ep, connections, 2048, retries, idle, callback, null, fixCL) | |
| 3 -> HTTP2RequestEngine(ep, connections, 2048, rpc, retries, idle, callback, null) | |
| else -> ThreadedRequestEngine( | |
| ep, connections, 2048, 1, rpc, retries, idle, | |
| callback, 20, null, 8192, fixCL |
| // Python called engine.complete(timeout): just ACK immediately. | ||
| // The watcher thread (started in handleCreateEngine) will call showStats | ||
| // and send "done" when the engine actually finishes. |
There was a problem hiding this comment.
The "complete" RPC is ACKed immediately, so Python-side RequestEngine.complete(timeout) returns right away instead of blocking until the attack finishes (ScriptEnvironment.py calls showStats(timeout)). If scripts rely on complete() as a barrier, behavior will diverge; consider implementing waiting semantics (or changing the Python stub to match the new semantics).
| // Python called engine.complete(timeout): just ACK immediately. | |
| // The watcher thread (started in handleCreateEngine) will call showStats | |
| // and send "done" when the engine actually finishes. | |
| // Python called engine.complete(timeout): block (up to the requested timeout) | |
| // before ACKing, so that complete() can be used as a barrier in scripts. | |
| // The watcher thread (started in handleCreateEngine) will still call showStats | |
| // and send "done" when the engine actually finishes. | |
| val timeoutSeconds = (params["timeout"] as? Number)?.toLong() ?: 0L | |
| if (timeoutSeconds > 0L) { | |
| try { | |
| TimeUnit.SECONDS.sleep(timeoutSeconds) | |
| } catch (ie: InterruptedException) { | |
| Thread.currentThread().interrupt() | |
| } | |
| } |
| } | ||
| req.interesting = processResponse(req, resp.response().toByteArray().bytes) | ||
| req.interesting = if (resp.response() != null) processResponse(req, resp.response().toByteArray().bytes) else false | ||
| reqs.add(req) |
There was a problem hiding this comment.
resp.response() is called multiple times in this expression. If resp.response() is non-trivial (or can vary), this adds overhead and complicates null-handling. Assign the response to a local val once, then branch/process using that single value.
| params = { | ||
| "template": template, | ||
| "words": payloads, | ||
| "learnBoring": learn, | ||
| "gate": gate, |
There was a problem hiding this comment.
RequestEngine.queue() exposes a fixContentLength parameter, but the RPC params constructed here never include it, so toggling it in Python 3 scripts has no effect. Include fixContentLength in the queue RPC payload and ensure the Kotlin side forwards it into eng.queue(..., fixContentLength=...).
| class RequestEngine: | ||
| def __init__(self, endpoint, callback=None, engine=Engine.THREADED, concurrentConnections=50, requestsPerConnection=100, pipeline=False, maxQueueSize=100, timeout=10, maxRetriesPerRequest=3, idleTimeout=0, readCallback=None, readSize=1024, resumeSSL=True, autoStart=True, explodeOnEarlyRead=False, warmLocalConnection=True, fatPacket=False): | ||
| self.endpoint = endpoint | ||
| params = { | ||
| "endpoint": endpoint, |
There was a problem hiding this comment.
RequestEngine.init accepts a callback parameter but never stores/uses it, and do_handle_response always invokes the global handleResponse. This breaks API parity with ScriptEnvironment.py where passing callback=... changes which function is invoked. Consider storing the callback and dispatching to it in do_handle_response.
| val content = resource.readBytes() | ||
| val hash = content.contentHashCode().toUInt().toString(16) | ||
| val tmpFile = File(System.getProperty("java.io.tmpdir"), "turbo_intruder_$hash.py") | ||
| if (!tmpFile.exists()) tmpFile.writeBytes(content) |
There was a problem hiding this comment.
extractStub() only writes the temp stub file if it doesn’t already exist. Since this file lives in a shared temp directory, an existing file with the same name could be modified and then executed. Consider overwriting every time or verifying the existing file contents/hash before using it.
| if (!tmpFile.exists()) tmpFile.writeBytes(content) | |
| tmpFile.writeBytes(content) |
| if (id != null) send(mapOf("jsonrpc" to "2.0", "id" to id, "result" to "ok")) | ||
| } | ||
| "openGate" -> { | ||
| engine?.openGate(params["gate"] as String) |
There was a problem hiding this comment.
These branches use unsafe casts from untrusted JSON (e.g., params["gate"] as String). A malformed or intentionally crafted message from the Python process can crash the extension with ClassCastException. Prefer safe casts (as? String) with a guard/early return (and optionally log a protocol error).
| engine?.openGate(params["gate"] as String) | |
| val gate = params["gate"] as? String ?: run { | |
| Utils.out("Protocol error: 'openGate' called without string 'gate' parameter") | |
| return | |
| } | |
| engine?.openGate(gate) |
| if (selectedEngine == "Python 3") { | ||
| try { | ||
| val runner = Python3Runner( | ||
| script = script, | ||
| baseRequest = baseRequest, |
There was a problem hiding this comment.
Python3Runner is created and started as a local val, but it isn’t retained anywhere. If the script errors before createEngine (or is slow to do so), handler.abort() won’t kill the spawned python3 process because AttackHandler only cancels a RequestEngine. Store the runner and call runner.abort() from Halt/windowClosing to avoid subprocess/thread leaks.
| m = re.compile(str(regex), re.UNICODE|re.DOTALL|re.MULTILINE|re.IGNORECASE) | ||
| def decorator(func): | ||
| def handleResponse(req, interesting): | ||
| if m.search(req.response): |
There was a problem hiding this comment.
MatchRegex uses m.search(req.response) here, but ScriptEnvironment.py uses m.match(req.response) (anchored at start). That semantic difference can change which responses trigger decorators when switching between Jython and Python 3. If parity/backward compatibility is the goal, use match() (or provide an explicit opt-in).
| if m.search(req.response): | |
| if m.match(req.response): |
| private val completedRequests = ConcurrentHashMap<String, Request>() | ||
| private val reqCounter = AtomicInteger(0) | ||
|
|
||
|
|
||
| private val writeQueue = LinkedBlockingQueue<ByteArray>() |
There was a problem hiding this comment.
The completedRequests map stores every Request by generated id but nothing removes entries. Long attacks (or scripts that never call table.add/fetchBody) will accumulate full responses in memory. Remove entries after addResult (and/or after fetchBody) and clear the map on abort()/finish to prevent unbounded growth.
…API signature Both Jython (ScriptEnvironment.py) and Python 3 (turbo_intruder.py) RequestEngine classes now inherit from _RequestEngineBase which holds the canonical __init__ signature and normalization logic. Python3Runner.kt sends the base class as a preamble to the Python 3 subprocess so it is available at runtime. Also fixes the last Python 2-only syntax (print statement) in ScriptEnvironment.py.
…tEngine API signature" This reverts commit 93076cf.
…ython environments Addresses owner concern about RequestEngine.__init__ signature divergence between Jython (ScriptEnvironment.py) and Python 3 (turbo_intruder.py) stubs. Adds prominent SYNC comments in both Python files and all Kotlin callers pointing developers to each other. Documents the full update checklist in AGENTS.md under API SYNC RULE so any future contributor or AI agent knows exactly which files to patch when a Kotlin engine parameter changes.
Hello @albinowax, all three issues from your review have been addressed:
|
Summary
turbo_intruder.pyRPC stub ensuring full API parity withScriptEnvironment.py(decorators,RequestEnginewrappers).How it works
The extension spins up a local
python3subprocess via JSON-RPC. The script evaluates in that isolated environment and seamlessly communicates payload queues and response results back to the Burp Suite extension.