Testing applications that use fsspec.open("s3://...") #1785

dionhaefner · 2025-01-30T11:35:31Z

dionhaefner
Jan 30, 2025

I'm trying to test an application that uses fsspec under the hood to open files passed in by users as s3://... strings. I thought I could use the moto proxy mode for this, but it doesn't seem to work (looks like fsspec is trying to hit live AWS instead, with a permissions error).

Helpful for any pointers on how to test S3 integration effectively.

Here's what I tried:

from moto import mock_aws
import json
import os

os.environ["TEST_PROXY_MODE"] = "true"

with mock_aws():
    import boto3
    import fsspec

    client = boto3.client("s3")
    client.create_bucket(Bucket="test-bucket")

    client.put_object(
        Bucket="test-bucket",
        Key="foo.json",
        Body=json.dumps({"inputs": "foo"}),
    )

    with fsspec.open("s3://test-bucket/foo.json") as f:
        print(f.read())

$ moto_proxy &
$ python foo.py
INFO 2025-01-30 12:31:34,189 - POST http://localhost:5005/moto-api/reset
/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
INFO 2025-01-30 12:31:34,329 - PUT https://test-bucket.s3.amazonaws.com/
/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
INFO 2025-01-30 12:31:34,404 - PUT https://test-bucket.s3.amazonaws.com/foo.json
Traceback (most recent call last):
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 114, in _error_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/aiobotocore/client.py", line 412, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dion/codes/tesseract/foo.py", line 20, in <module>
    with fsspec.open("s3://test-bucket/foo.json") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/core.py", line 105, in __enter__
    f = self.fs.open(self.path, mode=mode)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/spec.py", line 1310, in open
    f = self._open(
        ^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 699, in _open
    return S3File(
           ^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 2238, in __init__
    super().__init__(
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/spec.py", line 1884, in __init__
    self.size = self.details["size"]
                ^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/spec.py", line 1897, in details
    self._details = self.fs.info(self.path)
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 1426, in _info
    out = await self._call_s3(
          ^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 371, in _call_s3
    return await _error_wrapper(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 146, in _error_wrapper
    raise err
PermissionError: Forbidden

When not using proxy mode, I get this traceback instead:

Traceback (most recent call last):
  File "/Users/dion/codes/tesseract/foo.py", line 20, in <module>
    with fsspec.open("s3://test-bucket/foo.json") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/core.py", line 105, in __enter__
    f = self.fs.open(self.path, mode=mode)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/spec.py", line 1310, in open
    f = self._open(
        ^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 699, in _open
    return S3File(
           ^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 2238, in __init__
    super().__init__(
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/spec.py", line 1884, in __init__
    self.size = self.details["size"]
                ^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/spec.py", line 1897, in details
    self._details = self.fs.info(self.path)
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 1426, in _info
    out = await self._call_s3(
          ^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 371, in _call_s3
    return await _error_wrapper(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 146, in _error_wrapper
    raise err
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/s3fs/core.py", line 114, in _error_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/aiobotocore/client.py", line 394, in _make_api_call
    http, parsed_response = await self._make_request(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/aiobotocore/client.py", line 420, in _make_request
    return await self._endpoint.make_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/aiobotocore/endpoint.py", line 97, in _send_request
    success_response, exception = await self._get_response(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/aiobotocore/endpoint.py", line 139, in _get_response
    success_response, exception = await self._do_get_response(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/aiobotocore/endpoint.py", line 191, in _do_get_response
    response_dict = await convert_to_response_dict(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dion/.virtualenvs/science/lib/python3.12/site-packages/aiobotocore/endpoint.py", line 49, in convert_to_response_dict
    for k, v in http_response.raw.raw_headers
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'MockRawResponse' object has no attribute 'raw_headers'

Answered by dionhaefner

Jan 30, 2025

Example above is using mock_aws, which raises both with and without proxy mode (different errors tho).

Thanks for the pointer regarding global config though, that's helpful! This works now, even without monkeypatching anything:

import os
port = 5123
server_url = f"http://localhost:{port}"
os.environ["FSSPEC_S3_ENDPOINT_URL"] = server_url

import json
import boto3
import fsspec
from moto.server import ThreadedMotoServer

server = ThreadedMotoServer(port=port)
server.start()

client = boto3.client("s3", endpoint_url=server_url)
client.create_bucket(Bucket="test-bucket")

client.put_object(
    Bucket="test-bucket",
    Key="foo.json",
    Body=json.dumps({"inputs": "foo"}),
)

with fsspec.open

View full answer

martindurant · 2025-01-30T14:13:25Z

martindurant
Jan 30, 2025
Maintainer

The above LLM answer is essentially correct, but the key is the argument endpoint_url - you use this to tell s3fs which S3 to talk to.

Where did you encounter the environment variable TEST_PROXY_MODE, is this documented?

5 replies

dionhaefner Jan 30, 2025
Author

Believe it or not, this is actually not an LLM answer, just something I wrote myself 😉

the key is the argument endpoint_url - you use this to tell s3fs which S3 to talk to.

I don't have control over the fsspec.open call from the API I'm trying to test. I was hoping I could use moto proxy mode to intercept the calls to S3 and replace them with their mocked equivalent.

Where did you encounter the environment variable TEST_PROXY_MODE, is this documented?

It's from moto's proxy mode:

https://docs.getmoto.org/en/latest/docs/proxy_mode.html#python-configuration

martindurant Jan 30, 2025
Maintainer

It's from moto's proxy mode:

OK, then you will need to use mock_aws as a decorator or context manager. This isn't quite what the s3fs test suite does (it wants to be more explicit and control where moto runs), but I expect ti to work.

Alternatively, fsspec supports configuration allowing you to pass default arguments to filesystem instantiation. https://filesystem-spec.readthedocs.io/en/latest/features.html#configuration
A file in “~/.config/fsspec/conf.json” with {"s3": {"endpoint_url": "..."}} or FSSPEC_S3_ENDPOINT_URL=... should work.

martindurant Jan 30, 2025
Maintainer

Believe it or not, this is actually not an LLM answer, just something I wrote myself

Apologies; maybe we are all a bit LLM now.

dionhaefner Jan 30, 2025
Author

Example above is using mock_aws, which raises both with and without proxy mode (different errors tho).

Thanks for the pointer regarding global config though, that's helpful! This works now, even without monkeypatching anything:

import os
port = 5123
server_url = f"http://localhost:{port}"
os.environ["FSSPEC_S3_ENDPOINT_URL"] = server_url

import json
import boto3
import fsspec
from moto.server import ThreadedMotoServer

server = ThreadedMotoServer(port=port)
server.start()

client = boto3.client("s3", endpoint_url=server_url)
client.create_bucket(Bucket="test-bucket")

client.put_object(
    Bucket="test-bucket",
    Key="foo.json",
    Body=json.dumps({"inputs": "foo"}),
)

with fsspec.open("s3://test-bucket/foo.json") as f:
    print(f.read())

Answer selected by dionhaefner

martindurant Jan 30, 2025
Maintainer

Just to note that this:

AttributeError: 'MockRawResponse' object has no attribute 'raw_headers'

looks like it might be a version mismatch between moto and botocore, but I'm not sure.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing applications that use fsspec.open("s3://...") #1785

{{title}}

Replies: 1 comment 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Testing applications that use fsspec.open("s3://...") #1785

dionhaefner Jan 30, 2025

Replies: 1 comment · 5 replies

martindurant Jan 30, 2025 Maintainer

dionhaefner Jan 30, 2025 Author

martindurant Jan 30, 2025 Maintainer

martindurant Jan 30, 2025 Maintainer

dionhaefner Jan 30, 2025 Author

martindurant Jan 30, 2025 Maintainer

dionhaefner
Jan 30, 2025

Replies: 1 comment 5 replies

martindurant
Jan 30, 2025
Maintainer

dionhaefner Jan 30, 2025
Author

martindurant Jan 30, 2025
Maintainer

martindurant Jan 30, 2025
Maintainer

dionhaefner Jan 30, 2025
Author

martindurant Jan 30, 2025
Maintainer