`web.scrape` fails for some URLs while the API endpoint works

[ndhoanit1112]

I encountered an issue where supadata.web.scrape(url) fails for certain URLs, but when I call the API endpoint directly, it works fine.

Steps to Reproduce

from supadata import Supadata

supadata = Supadata(api_key="MY_API_KEY")

web_content = supadata.web.scrape("https://openai.com/index/introducing-deep-research/")

print(f"Page title: {web_content.name}")
print(f"Page content: {web_content.content}")

Expected Behavior

The function should return the page title and content, just like when calling the API directly.

Actual Behavior

The function fails for certain URLs, but the API works fine when tested manually.

Error Message

urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

SDK Version

1.0.4

Would appreciate any insights or fixes. Thanks!

[rafalzawadzki]

hi @ndhoanit1112 this looks like a timeout error. Could you try adding your own timeout to test?