Skip to content

Commit 359613f

Browse files
authored
feat: iplist2rule utility command (#1373)
* feat: iplist2rule utility command Assisted-By: GLM 4.7 via Claude Code Signed-off-by: Xe Iaso <me@xeiaso.net> * docs: update CHANGELOG Signed-off-by: Xe Iaso <me@xeiaso.net> * chore: fix spelling Signed-off-by: Xe Iaso <me@xeiaso.net> * chore: fix spelling again Signed-off-by: Xe Iaso <me@xeiaso.net> * feat(iplist2rule): add comment describing how rule was generated Signed-off-by: Xe Iaso <me@xeiaso.net> * docs: add iplist2rule docs Signed-off-by: Xe Iaso <me@xeiaso.net> * chore: fix spelling Signed-off-by: Xe Iaso <me@xeiaso.net> --------- Signed-off-by: Xe Iaso <me@xeiaso.net>
1 parent 1d8e98c commit 359613f

6 files changed

Lines changed: 223 additions & 4 deletions

File tree

.github/actions/spelling/allow.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,8 @@ clampip
1818
pseudoprofound
1919
reimagining
2020
iocaine
21+
admins
22+
fout
23+
iplist
24+
NArg
25+
blocklists

docs/docs/CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1111

1212
## [Unreleased]
1313

14+
- Add iplist2rule tool that lets admins turn an IP address blocklist into an Anubis ruleset.
1415
- Add Polish locale ([#1292](https://github.com/TecharoHQ/anubis/pull/1309))
1516

1617
<!-- This changes the project to: -->

docs/docs/admin/iplist2rule.mdx

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
---
2+
title: iplist2rule CLI tool
3+
---
4+
5+
The `iplist2rule` tool converts IP blocklists into Anubis challenge policies. It reads common IP block list formats and generates the appropriate Anubis policy file for IP address filtering.
6+
7+
## Installation
8+
9+
Install directly with Go
10+
11+
```bash
12+
go install github.com/TecharoHQ/anubis/utils/cmd/iplist2rule@latest
13+
```
14+
15+
## Usage
16+
17+
Basic conversion from URL:
18+
19+
```bash
20+
iplist2rule https://raw.githubusercontent.com/7c/torfilter/refs/heads/main/lists/txt/torfilter-1m-flat.txt filter-tor.yaml
21+
```
22+
23+
Explicitly allow every IP address on a list:
24+
25+
```bash
26+
iplist2rule --action ALLOW https://raw.githubusercontent.com/7c/torfilter/refs/heads/main/lists/txt/torfilter-1m-flat.txt filter-tor.yaml
27+
```
28+
29+
Add weight to requests matching IP addresses on a list:
30+
31+
```bash
32+
iplist2rule --action WEIGH --weight 20 https://raw.githubusercontent.com/7c/torfilter/refs/heads/main/lists/txt/torfilter-1m-flat.txt filter-tor.yaml
33+
```
34+
35+
## Options
36+
37+
| Flag | Description | Default |
38+
| :------------ | :----------------------------------------------------------------------------------------------- | :-------------------------------- |
39+
| `--action` | The Anubis action to take for the IP address in question, must be in ALL CAPS. | `DENY` (forbids traffic) |
40+
| `--rule-name` | The name for the generated Anubis rule, should be in kebab-case. | (not set, inferred from filename) |
41+
| `--weight` | When `--action=WEIGH`, how many weight points should be added or removed from matching requests? | 0 (not set) |
42+
43+
## Using the Generated Policy
44+
45+
Save the output and import it in your main policy file:
46+
47+
```yaml
48+
bots:
49+
- import: "./filter-tor.yaml"
50+
```

docs/docs/admin/robots2policy.mdx

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ Install directly with Go:
1212
```bash
1313
go install github.com/TecharoHQ/anubis/cmd/robots2policy@latest
1414
```
15+
1516
## Usage
1617

1718
Basic conversion from URL:
@@ -35,8 +36,8 @@ robots2policy -input robots.txt -action DENY -format json
3536
## Options
3637

3738
| Flag | Description | Default |
38-
|-----------------------|--------------------------------------------------------------------|---------------------|
39-
| `-input` | robots.txt file path or URL (use `-` for stdin) | *required* |
39+
| --------------------- | ------------------------------------------------------------------ | ------------------- |
40+
| `-input` | robots.txt file path or URL (use `-` for stdin) | _required_ |
4041
| `-output` | Output file (use `-` for stdout) | stdout |
4142
| `-format` | Output format: `yaml` or `json` | `yaml` |
4243
| `-action` | Action for disallowed paths: `ALLOW`, `DENY`, `CHALLENGE`, `WEIGH` | `CHALLENGE` |
@@ -47,6 +48,7 @@ robots2policy -input robots.txt -action DENY -format json
4748
## Example
4849

4950
Input robots.txt:
51+
5052
```txt
5153
User-agent: *
5254
Disallow: /admin/
@@ -57,6 +59,7 @@ Disallow: /
5759
```
5860

5961
Generated policy:
62+
6063
```yaml
6164
- name: robots-txt-policy-disallow-1
6265
action: CHALLENGE
@@ -77,8 +80,8 @@ Generated policy:
7780
Save the output and import it in your main policy file:
7881
7982
```yaml
80-
import:
81-
- path: "./robots-policy.yaml"
83+
bots:
84+
- import: "./robots-policy.yaml"
8285
```
8386
8487
The tool handles wildcard patterns, user-agent specific rules, and blacklisted bots automatically.

utils/cmd/iplist2rule/blocklist.go

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
package main
2+
3+
import (
4+
"bufio"
5+
"fmt"
6+
"io"
7+
"net/http"
8+
"net/netip"
9+
"strings"
10+
)
11+
12+
// FetchBlocklist reads the blocklist over HTTP and returns every non-commented
13+
// line parsed as an IP address in CIDR notation. IPv4 addresses are returned as
14+
// /32, IPv6 addresses as /128.
15+
//
16+
// This function was generated with GLM 4.7.
17+
func FetchBlocklist(url string) ([]string, error) {
18+
resp, err := http.Get(url)
19+
if err != nil {
20+
return nil, err
21+
}
22+
defer resp.Body.Close()
23+
24+
if resp.StatusCode != http.StatusOK {
25+
return nil, fmt.Errorf("HTTP request failed with status: %s", resp.Status)
26+
}
27+
28+
var lines []string
29+
scanner := bufio.NewScanner(resp.Body)
30+
for scanner.Scan() {
31+
line := scanner.Text()
32+
// Skip empty lines and comments (lines starting with #)
33+
if line == "" || strings.HasPrefix(line, "#") {
34+
continue
35+
}
36+
37+
addr, err := netip.ParseAddr(line)
38+
if err != nil {
39+
// Skip lines that aren't valid IP addresses
40+
continue
41+
}
42+
43+
var cidr string
44+
if addr.Is4() {
45+
cidr = fmt.Sprintf("%s/32", addr.String())
46+
} else {
47+
cidr = fmt.Sprintf("%s/128", addr.String())
48+
}
49+
lines = append(lines, cidr)
50+
}
51+
52+
if err := scanner.Err(); err != nil && err != io.EOF {
53+
return nil, err
54+
}
55+
56+
return lines, nil
57+
}

utils/cmd/iplist2rule/main.go

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
package main
2+
3+
import (
4+
"flag"
5+
"fmt"
6+
"log"
7+
"os"
8+
"path/filepath"
9+
"strings"
10+
"time"
11+
12+
"github.com/TecharoHQ/anubis/lib/config"
13+
"github.com/facebookgo/flagenv"
14+
"sigs.k8s.io/yaml"
15+
)
16+
17+
type Rule struct {
18+
Name string `yaml:"name" json:"name"`
19+
Action config.Rule `yaml:"action" json:"action"`
20+
RemoteAddr []string `json:"remote_addresses,omitempty" yaml:"remote_addresses,omitempty"`
21+
Weight *config.Weight `json:"weight,omitempty" yaml:"weight,omitempty"`
22+
}
23+
24+
func init() {
25+
flag.Usage = func() {
26+
fmt.Printf(`Usage of %[1]s:
27+
28+
%[1]s [flags] <blocklist-url> <filename>
29+
30+
Grabs the contents of the blocklist, converts it to an Anubis ruleset, and writes it to filename.
31+
32+
Flags:
33+
`, filepath.Base(os.Args[0]))
34+
35+
flag.PrintDefaults()
36+
}
37+
}
38+
39+
var (
40+
action = flag.String("action", "DENY", "Anubis action to take (ALLOW / DENY / WEIGH)")
41+
manualRuleName = flag.String("rule-name", "", "If set, prefer this name over inferring from filename")
42+
weight = flag.Int("weight", 0, "If set to any number, add/subtract this many weight points when --action=WEIGH")
43+
)
44+
45+
func main() {
46+
flagenv.Parse()
47+
flag.Parse()
48+
49+
if flag.NArg() != 2 {
50+
flag.Usage()
51+
os.Exit(2)
52+
}
53+
54+
blocklistURL := flag.Arg(0)
55+
foutName := flag.Arg(1)
56+
ruleName := strings.TrimSuffix(foutName, filepath.Ext(foutName))
57+
58+
if *manualRuleName != "" {
59+
ruleName = *manualRuleName
60+
}
61+
62+
ruleAction := config.Rule(*action)
63+
if err := ruleAction.Valid(); err != nil {
64+
log.Fatalf("--action=%q is invalid: %v", *action, err)
65+
}
66+
67+
result := &Rule{
68+
Name: ruleName,
69+
Action: ruleAction,
70+
}
71+
72+
if *weight != 0 {
73+
if ruleAction != config.RuleWeigh {
74+
log.Fatalf("used --weight=%d but --action=%s", *weight, *action)
75+
}
76+
77+
result.Weight = &config.Weight{
78+
Adjust: *weight,
79+
}
80+
}
81+
82+
ips, err := FetchBlocklist(blocklistURL)
83+
if err != nil {
84+
log.Fatalf("can't fetch blocklist %s: %v", blocklistURL, err)
85+
}
86+
87+
result.RemoteAddr = ips
88+
89+
fout, err := os.Create(foutName)
90+
if err != nil {
91+
log.Fatalf("can't create output file %q: %v", foutName, err)
92+
}
93+
defer fout.Close()
94+
95+
fmt.Fprintf(fout, "# Generated by %s on %s from %s\n\n", filepath.Base(os.Args[0]), time.Now().Format(time.RFC3339), blocklistURL)
96+
97+
data, err := yaml.Marshal([]*Rule{result})
98+
if err != nil {
99+
log.Fatalf("can't marshal yaml")
100+
}
101+
102+
fout.Write(data)
103+
}

0 commit comments

Comments
 (0)