feat: implement HTTP caching with mitmproxy's native format

[devin-ai-integration]

This PR implements HTTP caching for connectors using mitmproxy. It supports four cache modes, separate read/write directories, and configurable serialization formats including mitmproxy's native format for better interoperability.

Link to Devin run: https://app.devin.ai/sessions/9bbcc89c5dc047cabfe064370d8ca798
Requested by: Aaron ("AJ") Steers ([email protected])

[devin-ai-integration]

🤖 Devin AI Engineer

Original prompt from Aaron:

# Task: Implement HTTP Caching in PyAirbyte

## Background
We need to port the MITM (Man-In-The-Middle) proxy functionality from the Airbyte CI system into the PyAirbyte repository. This functionality allows for caching and replaying HTTP requests, which is useful for working around rate limits and situations where credentials might not be available.

## Requirements

### 1. Create a new module called `http_caching`
- Create a new module in the PyAirbyte repository called `http_caching` (separate from the existing `caches` module)
- The main class should be called `AirbyteConnectorCache`
- Implement the functionality using mitmproxy's Python API (not the command-line interface)

### 2. Configuration
- By default, cache files should be stored in a local directory called `.airbyte-http-cache`
- Create a constant that points to this directory
- Add support for an environment variable to override the cache location
- The primary means of sending traffic to this proxy should be via the HTTP_PROXY environment variable

### 3. Modify `get_source` implementation
- Add a new optional parameter called `http_cache` to the `get_source` function in `airbyte/sources/util.py`
- When this parameter is specified, configure the source to use the HTTP caching functionality

### 4. Core Functionality
The implementation should support:
- Recording HTTP traffic between connectors and sources
- Replaying requests to the same URL from a previous run
- Avoiding rate limiting issues
- Handling certificate management for HTTPS interception
- Proper handling of sensitive data in cached responses

## Reference Files
- `airbyte/sources/util.py` - Contains the `get_source` function that needs modification
- `airbyte/sources/registry.py` - Related to source registration
- `airbyte/__init__.py` - Main module initialization

## Notes
- This is separate from the existing data caching system in `airbyte.caches`
- Do not implement Docker wrapper functionality at this time
- Focus on making a clean Python-based implementation

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[aaronsteers]

For the binary format, you used pickle. I was expecting this to be a format that is natively readable by another tool. Please check Airbyte repo's live-tests implementation for the correct format to use here. I don't recall specifically, but you can take this opportunity to check there and migrate similarly here if possible.

Also, they used something like "hal" or similar for their JSON format. I don't know what that is, so please explain it as well (but don't migrate to it yet).

[aaronsteers]

TODO: We may need to (later, not here) implement a cache option for declarative execution. This runs through our same Python process, so we may either need to patch the requests library or have another implementation. Lmk if you have thoughts, but don't implement.

[devin-ai-integration]

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

To fix the problem, we should bind the socket to a specific interface instead of all interfaces. Since the function find_free_port is used to find an available port, we can bind it to localhost to limit the exposure. This change will ensure that the socket only accepts connections from the local machine, reducing the security risk.

[devin-ai-integration]

Closing due to inactivity for more than 7 days.