Cloudflare checking browser, Subscene query returning E503

[radek555]

Hi,
recently subgrab stopped working for me. Today I digged a bit into what's wrong and after enabling DEBUG logging level, I got:

$ subgrab -m Titanic
DEBUG:SubGrab:Input with flags: ['subgrab', '-m', 'Titanic']
INFO:SubGrab:Initialized SubGrab script
INFO:SubGrab:Set Language: EN
INFO:SubGrab:Searching For: Titanic
INFO:subscene.py:Selecting title for name: Titanic
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): subscene.com:443
DEBUG:urllib3.connectionpool:https://subscene.com:443 "GET /subtitles/searchbytitle?query=Titanic HTTP/1.1" 503 None
DEBUG:subscene.py:https://subscene.com/subtitles/searchbytitle?query=Titanic not retrieved.
INFO:subscene.py:Searching in query: https://subscene.com/subtitles/searchbytitle/?query=Titanic
DEBUG:subscene.py:Returning - 'NoneType' object has no attribute 'find'
INFO:SubGrab:Subtitle Link for Titanic : None

I guess this is due to some protection mechanisms being enforced by subscene server, right? This is also evident when you go to subscene.com via browser, before loading the webpage, there is a banner saying

Checking your browser before accessing subscene.com.
This process is automatic. Your browser will redirect to your requested content shortly.
Please allow up to 5 seconds…
DDoS protection by Cloudflare

Is anyone going to fix this somehow and is that even possible?
Also, is it possible to use ALLSUBDB backend without actually computing the movie hash from media file, only using movie title?

Thanks!

[RafayGhafoor]

Thank you for notifying and this looks bad 😢 but we have a way to mitigate this issue. 😃

A workaround would be to use selenium and open a web-browser using your profile and save cookies obtained from subscene site. These cookies would be then read and set by the requests session for making requests to the site.

@radek555, Code snippet which I just tested to see if the browser allows access to subscene with my current profile is as follows and this would give you some pointers.

Subscene Cookies

I would be happy to guide you on this issue and would appreciate PR that could add this patch to subscene library.

[radek555]

Hi,
thanks for the quick reply! I tried with selenium to get the cookies and then adding these cookies to requests.get(url, cookies=cookies) and it worked.

Unfortunately I don't have more time to fix this issue. I am sorry. Also I don't know how the integration of this selenium trick should look like in wrt. subgrab (command line argument, config file?). Maybe if you describe how this should be carried out, someone else can help.

[RafayGhafoor]

I will try to fix this issue when I have time as it's not persistent for everybody and the fix would introduce additional dependencies in the project :).

[nakhla]

Hi, thanks for the quick reply! I tried with selenium to get the cookies and then adding these cookies to requests.get(url, cookies=cookies) and it worked.

Unfortunately I don't have more time to fix this issue. I am sorry. Also I don't know how the integration of this selenium trick should look like in wrt. subgrab (command line argument, config file?). Maybe if you describe how this should be carried out, someone else can help.

I am still getting Unable to make request. Status Code 403 even after fetching the cookies and adding them in the request, any idea to overcome this issue.
my simple project is also affected by this limitation from subscene. any idea is appreciated

[RafayGhafoor]

please post your code which is trying to make request using cookies so I can check what could be done :)