Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions data/botPolicies.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,7 @@ bots:
# user_agent_regex: (?i:bot|crawler)
# action: CHALLENGE
# challenge:
# difficulty: 16 # impossible
# report_as: 4 # lie to the operator
# difficulty: 16 # impossible
# algorithm: slow # intentionally waste CPU cycles and time

# Requires a subscription to Thoth to use, see
Expand Down Expand Up @@ -249,7 +248,6 @@ thresholds:
# https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh
algorithm: metarefresh
difficulty: 1
report_as: 1
# For clients that are browser-like but have either gained points from custom rules or
# report as a standard browser.
- name: moderate-suspicion
Expand All @@ -262,7 +260,6 @@ thresholds:
# https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
algorithm: fast
difficulty: 2 # two leading zeros, very fast for most clients
report_as: 2
- name: mild-proof-of-work
expression:
all:
Expand All @@ -273,7 +270,6 @@ thresholds:
# https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
algorithm: fast
difficulty: 4
report_as: 4
# For clients that are browser like and have gained many points from custom rules
- name: extreme-suspicion
expression: weight >= 30
Expand All @@ -282,4 +278,3 @@ thresholds:
# https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
algorithm: fast
difficulty: 6
report_as: 6
1 change: 0 additions & 1 deletion data/meta/default-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@
# action: CHALLENGE
# challenge:
# difficulty: 16 # impossible
# report_as: 4 # lie to the operator
# algorithm: slow # intentionally waste CPU cycles and time

# Requires a subscription to Thoth to use, see
Expand Down
21 changes: 21 additions & 0 deletions docs/docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,27 @@
- Stabilize the CVE-2025-24369 regression test by always submitting an invalid proof instead of relying on random POW failures.
- Add Polish locale ([#1292](https://github.com/TecharoHQ/anubis/pull/1309))

### Deprecate `report_as` in challenge configuration

Previously Anubis let you lie to users about the difficulty of a challenge to interfere with operators of malicious scrapers as a psyops attack:
Comment thread Fixed

```yaml
bots:
# Punish any bot with "bot" in the user-agent string
# This is known to have a high false-positive rate, use at your own risk
- name: generic-bot-catchall
user_agent_regex: (?i:bot|crawler)
action: CHALLENGE
challenge:
difficulty: 16 # impossible
report_as: 4 # lie to the operator
algorithm: slow # intentionally waste CPU cycles and time
```

This has turned out to be a bad idea and has been removed.

If you are using this setting, you will get a warning in your logs. To remove this warning, remove this setting from your policy file.

### Logging customization

Anubis now supports the ability to log to multiple backends ("sinks"). This allows you to have Anubis [log to a file](./admin/policies.mdx#file-sink) instead of just logging to standard out. You can also customize the [logging level](./admin/policies.mdx#log-levels) in the policy file:
Expand Down
1 change: 0 additions & 1 deletion docs/docs/admin/configuration/challenges/metarefresh.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ To use it in your Anubis configuration:
action: CHALLENGE
challenge:
difficulty: 1 # Number of seconds to wait before refreshing the page
report_as: 4 # Unused by this challenge method
algorithm: metarefresh # Specify a non-JS challenge method
```

Expand Down
1 change: 0 additions & 1 deletion docs/docs/admin/configuration/challenges/preact.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ To use it in your Anubis configuration:
action: CHALLENGE
challenge:
difficulty: 1 # Number of seconds to wait before refreshing the page
report_as: 4 # Unused by this challenge method
algorithm: preact
```

Expand Down
3 changes: 0 additions & 3 deletions docs/docs/admin/configuration/thresholds.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ thresholds:
challenge:
algorithm: metarefresh
difficulty: 1
report_as: 1

- name: moderate-suspicion
expression:
Expand All @@ -52,15 +51,13 @@ thresholds:
challenge:
algorithm: fast
difficulty: 2
report_as: 2

- name: extreme-suspicion
expression: weight >= 20
action: CHALLENGE
challenge:
algorithm: fast
difficulty: 4
report_as: 4
```

This defines a suite of 4 thresholds:
Expand Down
2 changes: 0 additions & 2 deletions docs/docs/admin/policies.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,6 @@ This rule has been known to have a high false positive rate in testing. Please u
action: CHALLENGE
challenge:
difficulty: 16 # impossible
report_as: 4 # lie to the operator
algorithm: slow # intentionally waste CPU cycles and time
```

Expand All @@ -93,7 +92,6 @@ Challenges can be configured with these settings:
| Key | Example | Description |
| :----------- | :------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `difficulty` | `4` | The challenge difficulty (number of leading zeros) for proof-of-work. See [Why does Anubis use Proof-of-Work?](/docs/design/why-proof-of-work) for more details. |
| `report_as` | `4` | What difficulty the UI should report to the user. Useful for messing with industrial-scale scraping efforts. |
| `algorithm` | `"fast"` | The challenge method to use. See [the list of challenge methods](./configuration/challenges/) for more information. |

### Remote IP based filtering
Expand Down
5 changes: 0 additions & 5 deletions docs/manifest/cfg/anubis/botPolicies.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,6 @@ bots:
# action: CHALLENGE
# challenge:
# difficulty: 16 # impossible
# report_as: 4 # lie to the operator
# algorithm: slow # intentionally waste CPU cycles and time

- name: rss-feed-blog
Expand Down Expand Up @@ -105,7 +104,6 @@ thresholds:
# https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh
algorithm: metarefresh
difficulty: 1
report_as: 1
# For clients that are browser-like but have either gained points from custom rules or
# report as a standard browser.
- name: moderate-suspicion
Expand All @@ -122,7 +120,6 @@ thresholds:
# challenge data, and forwards that to the client.
algorithm: preact
difficulty: 1
report_as: 1
- name: mild-proof-of-work
expression:
all:
Expand All @@ -133,7 +130,6 @@ thresholds:
# https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
algorithm: fast
difficulty: 2 # two leading zeros, very fast for most clients
report_as: 2
# For clients that are browser like and have gained many points from custom rules
- name: extreme-suspicion
expression: weight >= 30
Expand All @@ -142,7 +138,6 @@ thresholds:
# https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
algorithm: fast
difficulty: 4
report_as: 4

dnsbl: false

Expand Down
5 changes: 2 additions & 3 deletions lib/anubis.go
Original file line number Diff line number Diff line change
Expand Up @@ -167,8 +167,8 @@ func (s *Server) hydrateChallengeRule(rule *policy.Bot, chall *challenge.Challen
if rule.Challenge.Difficulty == 0 {
rule.Challenge.Difficulty = chall.Difficulty
}
if rule.Challenge.ReportAs == 0 {
rule.Challenge.ReportAs = chall.Difficulty
if rule.Challenge.ReportAs != 0 {
s.logger.Warn("[DEPRECATION] the report_as field in this bot rule is deprecated, see https://github.com/TecharoHQ/anubis/issues/1310 for more information", "bot_name", rule.Name, "difficulty", rule.Challenge.Difficulty, "report_as", rule.Challenge.ReportAs)
}
if rule.Challenge.Algorithm == "" {
rule.Challenge.Algorithm = chall.Method
Expand Down Expand Up @@ -648,7 +648,6 @@ func (s *Server) check(r *http.Request, lg *slog.Logger) (policy.CheckResult, *p
return cr("default/allow", config.RuleAllow, weight), &policy.Bot{
Challenge: &config.ChallengeRules{
Difficulty: s.policy.DefaultDifficulty,
ReportAs: s.policy.DefaultDifficulty,
Algorithm: config.DefaultAlgorithm,
},
Rules: &checker.List{},
Expand Down
4 changes: 0 additions & 4 deletions lib/anubis_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -464,10 +464,6 @@ func TestCheckDefaultDifficultyMatchesPolicy(t *testing.T) {
if bot.Challenge.Difficulty != i {
t.Errorf("Challenge.Difficulty is wrong, wanted %d, got: %d", i, bot.Challenge.Difficulty)
}

if bot.Challenge.ReportAs != i {
t.Errorf("Challenge.ReportAs is wrong, wanted %d, got: %d", i, bot.Challenge.ReportAs)
}
})
}
}
Expand Down
1 change: 0 additions & 1 deletion lib/challenge/proofofwork/proofofwork_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ func TestBasic(t *testing.T) {
Challenge: &config.ChallengeRules{
Algorithm: "fast",
Difficulty: 0,
ReportAs: 0,
},
}
const challengeStr = "hunter"
Expand Down
3 changes: 0 additions & 3 deletions lib/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,6 @@ func TestBotValid(t *testing.T) {
PathRegex: p("Mozilla"),
Challenge: &ChallengeRules{
Difficulty: -1,
ReportAs: 4,
Algorithm: "fast",
},
},
Expand All @@ -124,7 +123,6 @@ func TestBotValid(t *testing.T) {
PathRegex: p("Mozilla"),
Challenge: &ChallengeRules{
Difficulty: 420,
ReportAs: 4,
Algorithm: "fast",
},
},
Expand Down Expand Up @@ -361,7 +359,6 @@ func TestBotConfigZero(t *testing.T) {

b.Challenge = &ChallengeRules{
Difficulty: 4,
ReportAs: 4,
Algorithm: DefaultAlgorithm,
}
if b.Zero() {
Expand Down
3 changes: 0 additions & 3 deletions lib/config/testdata/good/thresholds.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ thresholds:
challenge:
algorithm: metarefresh
difficulty: 1
report_as: 1
- name: moderate-suspicion
expression:
all:
Expand All @@ -28,11 +27,9 @@ thresholds:
challenge:
algorithm: fast
difficulty: 2
report_as: 2
- name: extreme-suspicion
expression: weight >= 20
action: CHALLENGE
challenge:
algorithm: fast
difficulty: 4
report_as: 4
1 change: 0 additions & 1 deletion lib/config/threshold.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ var (
Challenge: &ChallengeRules{
Algorithm: "fast",
Difficulty: anubis.DefaultDifficulty,
ReportAs: anubis.DefaultDifficulty,
},
},
}
Expand Down
1 change: 0 additions & 1 deletion lib/config/threshold_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ func TestThresholdValid(t *testing.T) {
Challenge: &ChallengeRules{
Algorithm: "fast",
Difficulty: 1,
ReportAs: 1,
},
},
err: nil,
Expand Down
1 change: 0 additions & 1 deletion lib/policy/policy.go
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,6 @@ func ParseConfig(ctx context.Context, fin io.Reader, fname string, defaultDiffic
if b.Challenge == nil {
parsedBot.Challenge = &config.ChallengeRules{
Difficulty: defaultDifficulty,
ReportAs: defaultDifficulty,
Algorithm: "fast",
}
} else {
Expand Down
1 change: 0 additions & 1 deletion lib/testdata/invalid-challenge-method.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,4 @@ bots:
action: CHALLENGE
challenge:
difficulty: 16
report_as: 4
algorithm: hunter2 # invalid algorithm
1 change: 0 additions & 1 deletion lib/testdata/test_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,3 @@ thresholds:
challenge:
algorithm: fast
difficulty: 1
report_as: 1
1 change: 0 additions & 1 deletion lib/testdata/zero_difficulty.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,3 @@ thresholds:
challenge:
algorithm: fast
difficulty: 0
report_as: 0
1 change: 0 additions & 1 deletion test/palemoon/anubis/anubis.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ bots:
action: CHALLENGE
challenge:
difficulty: 2
report_as: 2
algorithm: fast

status_codes:
Expand Down
4 changes: 2 additions & 2 deletions web/js/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ const t = (key) => translations[`js_${key}`] || translations[key] || key;
return;
}

status.innerHTML = `${t('calculating_difficulty')} ${rules.report_as}, `;
status.innerHTML = `${t('calculating_difficulty')} ${rules.difficulty}, `;
progress.style.display = "inline-block";

// the whole text, including "Speed:", as a single node, because some browsers
Expand All @@ -166,7 +166,7 @@ const t = (key) => translations[`js_${key}`] || translations[key] || key;

let lastSpeedUpdate = 0;
let showingApology = false;
const likelihood = Math.pow(16, -rules.report_as);
const likelihood = Math.pow(16, -rules.difficulty);

try {
const t0 = Date.now();
Expand Down
Loading