Bug Summary
The Anubis WAF blocks miniflux's default user agent
Description
Websites protected by Anubis reject miniflux by default if the path to the feed doesn't end in xml, rss or atom. This is the case for https://journals.openedition.org/ or https://thereader.mitpress.mit.edu/feed/.
Steps to Reproduce
- add a new feed with this URL: https://journals.openedition.org/?page=backend_rssbreves&rub=5562
- set the user agent to anything that does not contain "Mozilla" or "Opera"
- the feed is successfully added
- remove the custom user agent in the feed's settings
- force refresh
Expected Behavior
The feed should continue updating normally.
Actual Behavior
Get the error "unable to detect feed format".
Version
2.3.2
Additional Context
The cause can be found at the bottom of anubis's default rules for what acts like a browser
# Generic catchall rule
- name: generic-browser
user_agent_regex: >-
Mozilla|Opera
action: WEIGH
weight:
adjust: 10I know in theory it is up to admins to configure Anubis correctly, like said in #3340. But they may be unreachable, slow to respond, and they may not care enough about RSS feeds. I've been messaging site admins left and right, and i'm always the first to report issues, even as they've been going on for months; this shows how little use and care they receive.
Even if Anubis was modified to better account for this issue, the new version could take a lot of time to reach affected websites.
It may be argued that the default user agent is already an attempt to get around restrictions. Indeed it has nothing to do with "Mozilla", and is just imitating browsers. Nowadays though, doing so is becoming a red flag. Miniflux should account for that.
Checklist
Bug Summary
The Anubis WAF blocks miniflux's default user agent
Description
Websites protected by Anubis reject miniflux by default if the path to the feed doesn't end in
xml,rssoratom. This is the case for https://journals.openedition.org/ or https://thereader.mitpress.mit.edu/feed/.Steps to Reproduce
Expected Behavior
The feed should continue updating normally.
Actual Behavior
Get the error "unable to detect feed format".
Version
2.3.2
Additional Context
The cause can be found at the bottom of anubis's default rules for what acts like a browser
I know in theory it is up to admins to configure Anubis correctly, like said in #3340. But they may be unreachable, slow to respond, and they may not care enough about RSS feeds. I've been messaging site admins left and right, and i'm always the first to report issues, even as they've been going on for months; this shows how little use and care they receive.
Even if Anubis was modified to better account for this issue, the new version could take a lot of time to reach affected websites.
It may be argued that the default user agent is already an attempt to get around restrictions. Indeed it has nothing to do with "Mozilla", and is just imitating browsers. Nowadays though, doing so is becoming a red flag. Miniflux should account for that.
Checklist