feat: implement HTTP caching with mitmproxy's native format#646
feat: implement HTTP caching with mitmproxy's native format#646devin-ai-integration[bot] wants to merge 21 commits intomainfrom
Conversation
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
🤖 Devin AI EngineerOriginal prompt from Aaron: I'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
Aaron ("AJ") Steers (aaronsteers)
left a comment
There was a problem hiding this comment.
For the binary format, you used pickle. I was expecting this to be a format that is natively readable by another tool. Please check Airbyte repo's live-tests implementation for the correct format to use here. I don't recall specifically, but you can take this opportunity to check there and migrate similarly here if possible.
Also, they used something like "hal" or similar for their JSON format. I don't know what that is, so please explain it as well (but don't migrate to it yet).
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
airbyte/_executors/declarative.py
Outdated
| ) -> Iterator[str]: | ||
| """Execute the declarative source.""" | ||
| _ = stdin # Not used | ||
| _ = stdin, env # Not used |
There was a problem hiding this comment.
TODO: We may need to (later, not here) implement a cache option for declarative execution. This runs through our same Python process, so we may either need to patch the requests library or have another implementation. Lmk if you have thoughts, but don't implement.
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
|
Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it. |
…mentation (#650) Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Aaron <AJ> Steers <aj@airbyte.io>
| def find_free_port() -> int: | ||
| """Find a free port to use for the proxy.""" | ||
| with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: | ||
| s.bind(("", 0)) |
Check warning
Code scanning / CodeQL
Binding a socket to all network interfaces Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 11 months ago
To fix the problem, we should bind the socket to a specific interface instead of all interfaces. Since the function find_free_port is used to find an available port, we can bind it to localhost to limit the exposure. This change will ensure that the socket only accepts connections from the local machine, reducing the security risk.
| @@ -33,3 +33,3 @@ | ||
| with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: | ||
| s.bind(("", 0)) | ||
| s.bind(("localhost", 0)) | ||
| return s.getsockname()[1] |
|
Closing due to inactivity for more than 7 days. |
This PR implements HTTP caching for connectors using mitmproxy. It supports four cache modes, separate read/write directories, and configurable serialization formats including mitmproxy's native format for better interoperability.
Link to Devin run: https://app.devin.ai/sessions/9bbcc89c5dc047cabfe064370d8ca798
Requested by: Aaron ("AJ") Steers (aj@airbyte.io)