Skip to content

lib/anubis: support setting extended cookie flags #120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Xe merged 3 commits into from
Mar 26, 2025
Merged

lib/anubis: support setting extended cookie flags #120

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e8dc037
lib/anubis: support setting extended cookie flags
Xe Mar 25, 2025
a57f2b3
lib: use cookie name consistently
Xe Mar 26, 2025
3da90ff
Merge branch 'main' into Xe/cookie-settings
Xe Mar 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions cmd/anubis/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ import (
var (
bind = flag.String("bind", ":8923", "network address to bind HTTP to")
bindNetwork = flag.String("bind-network", "tcp", "network family to bind HTTP to, e.g. unix, tcp")
cookieDomain = flag.String("cookie-domain", "", "if set, the top-level domain that the Anubis cookie will be valid for")
cookieName = flag.String("cookie-name", anubis.CookieName, "the name of the cookie that Anubis stores challenge pass records in")
cookiePartitioned = flag.Bool("cookie-partitioned", false, "if true, sets the partitioned flag on Anubis cookies, enabling CHIPS support")
challengeDifficulty = flag.Int("difficulty", anubis.DefaultDifficulty, "difficulty of the challenge")
ed25519PrivateKeyHex = flag.String("ed25519-private-key-hex", "", "private key used to sign JWTs, if not set a random one will be assigned")
metricsBind = flag.String("metrics-bind", ":9090", "network address to bind metrics to")
Expand Down
1 change: 1 addition & 0 deletions docs/docs/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Fixed bot check to only apply if address range matches
- Fix default difficulty setting that was broken in a refactor
- Linting fixes
- Extra cookie options can be set such as the name, domain, and partitioned flag [#73](https://github.com/TecharoHQ/anubis/issues/73)

## v1.14.2

Expand Down
27 changes: 15 additions & 12 deletions docs/docs/admin/installation.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,18 +41,21 @@ Anubis has very minimal system requirements. I suspect that 128Mi of ram may be

Anubis uses these environment variables for configuration:

| Environment Variable | Default value | Explanation |
| :------------------------ | :---------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BIND` | `:8923` | The network address that Anubis listens on. For `unix`, set this to a path: `/run/anubis/instance.sock` |
| `BIND_NETWORK` | `tcp` | The address family that Anubis listens on. Accepts `tcp`, `unix` and anything Go's [`net.Listen`](https://pkg.go.dev/net#Listen) supports. |
| `DIFFICULTY` | `5` | The difficulty of the challenge, or the number of leading zeroes that must be in successful responses. |
| `ED25519_PRIVATE_KEY_HEX` | | The hex-encoded ed25519 private key used to sign Anubis responses. If this is not set, Anubis will generate one for you. This should be exactly 64 characters long. See below for details. |
| `METRICS_BIND` | `:9090` | The network address that Anubis serves Prometheus metrics on. See `BIND` for more information. |
| `METRICS_BIND_NETWORK` | `tcp` | The address family that the Anubis metrics server listens on. See `BIND_NETWORK` for more information. |
| `SOCKET_MODE` | `0770` | _Only used when at least one of the `*_BIND_NETWORK` variables are set to `unix`._ The socket mode (permissions) for Unix domain sockets. |
| `POLICY_FNAME` | unset | The file containing [bot policy configuration](./policies.md). See the bot policy documentation for more details. If unset, the default bot policy configuration is used. |
| `SERVE_ROBOTS_TXT` | `false` | If set `true`, Anubis will serve a default `robots.txt` file that disallows all known AI scrapers by name and then additionally disallows every scraper. This is useful if facts and circumstances make it difficult to change the underlying service to serve such a `robots.txt` file. |
| `TARGET` | `http://localhost:3923` | The URL of the service that Anubis should forward valid requests to. Supports Unix domain sockets, set this to a URI like so: `unix:///path/to/socket.sock`. |
| Environment Variable | Default value | Explanation |
| :------------------------ | :--------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BIND` | `:8923` | The network address that Anubis listens on. For `unix`, set this to a path: `/run/anubis/instance.sock` |
| `BIND_NETWORK` | `tcp` | The address family that Anubis listens on. Accepts `tcp`, `unix` and anything Go's [`net.Listen`](https://pkg.go.dev/net#Listen) supports. |
| `COOKIE_DOMAIN` | unset | The domain the Anubis challenge pass cookie should be set to. This should be set to the domain you bought from your registrar (EG: `techaro.lol` if your webapp is running on `anubis.techaro.lol`). See [here](https://stackoverflow.com/a/1063760) for more information. |
Copy link
Copy Markdown
Contributor

@SuperSandro2000 SuperSandro2000 Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to mention that under all domains the same anubis or the same secrets should be used?

| `COOKIE_NAME` | `within.website-x-cmd-anubis-auth` | The cookie that Anubis uses to determine if users have passed a challenge. This should not be changed without a good reason. |
| `COOKIE_PARTITIONED` | `false` | If set to `true`, enables the [partitioned (CHIPS) flag](https://developers.google.com/privacy-sandbox/cookies/chips), meaning that Anubis inside an iframe has a different set of cookies than the domain hosting the iframe. |
| `DIFFICULTY` | `5` | The difficulty of the challenge, or the number of leading zeroes that must be in successful responses. |
| `ED25519_PRIVATE_KEY_HEX` | unset | The hex-encoded ed25519 private key used to sign Anubis responses. If this is not set, Anubis will generate one for you. This should be exactly 64 characters long. See below for details. |
| `METRICS_BIND` | `:9090` | The network address that Anubis serves Prometheus metrics on. See `BIND` for more information. |
| `METRICS_BIND_NETWORK` | `tcp` | The address family that the Anubis metrics server listens on. See `BIND_NETWORK` for more information. |
| `SOCKET_MODE` | `0770` | _Only used when at least one of the `*_BIND_NETWORK` variables are set to `unix`._ The socket mode (permissions) for Unix domain sockets. |
| `POLICY_FNAME` | unset | The file containing [bot policy configuration](./policies.md). See the bot policy documentation for more details. If unset, the default bot policy configuration is used. |
| `SERVE_ROBOTS_TXT` | `false` | If set `true`, Anubis will serve a default `robots.txt` file that disallows all known AI scrapers by name and then additionally disallows every scraper. This is useful if facts and circumstances make it difficult to change the underlying service to serve such a `robots.txt` file. |
| `TARGET` | `http://localhost:3923` | The URL of the service that Anubis should forward valid requests to. Supports Unix domain sockets, set this to a URI like so: `unix:///path/to/socket.sock`. |

### Key generation

Expand Down
49 changes: 28 additions & 21 deletions lib/anubis.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ type Options struct {
Policy *policy.ParsedConfig
ServeRobotsTXT bool
PrivateKey ed25519.PrivateKey

CookieDomain string
CookieName string
CookiePartitioned bool
}

func LoadPoliciesOrDefault(fname string, defaultDifficulty int) (*policy.ParsedConfig, error) {
Expand Down Expand Up @@ -108,6 +112,7 @@ func New(opts Options) (*Server, error) {
priv: opts.PrivateKey,
pub: opts.PrivateKey.Public().(ed25519.PublicKey),
policy: opts.Policy,
opts: opts,
DNSBLCache: decaymap.New[string, dnsbl.DroneBLResponse](),
}

Expand Down Expand Up @@ -145,6 +150,7 @@ type Server struct {
priv ed25519.PrivateKey
pub ed25519.PublicKey
policy *policy.ParsedConfig
opts Options
DNSBLCache *decaymap.Impl[string, dnsbl.DroneBLResponse]
ChallengeDifficulty int
}
Expand Down Expand Up @@ -217,7 +223,7 @@ func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {
s.next.ServeHTTP(w, r)
return
case config.RuleDeny:
ClearCookie(w)
s.ClearCookie(w)
lg.Info("explicit deny")
if rule == nil {
lg.Error("rule is nil, cannot calculate checksum")
Expand All @@ -236,29 +242,29 @@ func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {
case config.RuleChallenge:
lg.Debug("challenge requested")
default:
ClearCookie(w)
s.ClearCookie(w)
templ.Handler(web.Base("Oh noes!", web.ErrorPage("Other internal server error (contact the admin)")), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
return
}

ckie, err := r.Cookie(anubis.CookieName)
if err != nil {
lg.Debug("cookie not found", "path", r.URL.Path)
ClearCookie(w)
s.ClearCookie(w)
s.RenderIndex(w, r)
return
}

if err := ckie.Valid(); err != nil {
lg.Debug("cookie is invalid", "err", err)
ClearCookie(w)
s.ClearCookie(w)
s.RenderIndex(w, r)
return
}

if time.Now().After(ckie.Expires) && !ckie.Expires.IsZero() {
lg.Debug("cookie expired", "path", r.URL.Path)
ClearCookie(w)
s.ClearCookie(w)
s.RenderIndex(w, r)
return
}
Expand All @@ -269,7 +275,7 @@ func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {

if err != nil || !token.Valid {
lg.Debug("invalid token", "path", r.URL.Path, "err", err)
ClearCookie(w)
s.ClearCookie(w)
s.RenderIndex(w, r)
return
}
Expand All @@ -284,15 +290,15 @@ func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {
claims, ok := token.Claims.(jwt.MapClaims)
if !ok {
lg.Debug("invalid token claims type", "path", r.URL.Path)
ClearCookie(w)
s.ClearCookie(w)
s.RenderIndex(w, r)
return
}
challenge := s.challengeFor(r, rule.Challenge.Difficulty)

if claims["challenge"] != challenge {
lg.Debug("invalid challenge", "path", r.URL.Path)
ClearCookie(w)
s.ClearCookie(w)
s.RenderIndex(w, r)
return
}
Expand All @@ -309,7 +315,7 @@ func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {
if subtle.ConstantTimeCompare([]byte(claims["response"].(string)), []byte(calculated)) != 1 {
lg.Debug("invalid response", "path", r.URL.Path)
failedValidations.Inc()
ClearCookie(w)
s.ClearCookie(w)
s.RenderIndex(w, r)
return
}
Expand Down Expand Up @@ -372,23 +378,23 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {

nonceStr := r.FormValue("nonce")
if nonceStr == "" {
ClearCookie(w)
s.ClearCookie(w)
lg.Debug("no nonce")
templ.Handler(web.Base("Oh noes!", web.ErrorPage("missing nonce")), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
return
}

elapsedTimeStr := r.FormValue("elapsedTime")
if elapsedTimeStr == "" {
ClearCookie(w)
s.ClearCookie(w)
lg.Debug("no elapsedTime")
templ.Handler(web.Base("Oh noes!", web.ErrorPage("missing elapsedTime")), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
return
}

elapsedTime, err := strconv.ParseFloat(elapsedTimeStr, 64)
if err != nil {
ClearCookie(w)
s.ClearCookie(w)
lg.Debug("elapsedTime doesn't parse", "err", err)
templ.Handler(web.Base("Oh noes!", web.ErrorPage("invalid elapsedTime")), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
return
Expand All @@ -404,7 +410,7 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {

nonce, err := strconv.Atoi(nonceStr)
if err != nil {
ClearCookie(w)
s.ClearCookie(w)
lg.Debug("nonce doesn't parse", "err", err)
templ.Handler(web.Base("Oh noes!", web.ErrorPage("invalid nonce")), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
return
Expand All @@ -414,7 +420,7 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {
calculated := internal.SHA256sum(calcString)

if subtle.ConstantTimeCompare([]byte(response), []byte(calculated)) != 1 {
ClearCookie(w)
s.ClearCookie(w)
lg.Debug("hash does not match", "got", response, "want", calculated)
templ.Handler(web.Base("Oh noes!", web.ErrorPage("invalid response")), templ.WithStatus(http.StatusForbidden)).ServeHTTP(w, r)
failedValidations.Inc()
Expand All @@ -423,7 +429,7 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {

// compare the leading zeroes
if !strings.HasPrefix(response, strings.Repeat("0", s.ChallengeDifficulty)) {
ClearCookie(w)
s.ClearCookie(w)
lg.Debug("difficulty check failed", "response", response, "difficulty", s.ChallengeDifficulty)
templ.Handler(web.Base("Oh noes!", web.ErrorPage("invalid response")), templ.WithStatus(http.StatusForbidden)).ServeHTTP(w, r)
failedValidations.Inc()
Expand All @@ -442,17 +448,18 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {
tokenString, err := token.SignedString(s.priv)
if err != nil {
lg.Error("failed to sign JWT", "err", err)
ClearCookie(w)
s.ClearCookie(w)
templ.Handler(web.Base("Oh noes!", web.ErrorPage("failed to sign JWT")), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
return
}

http.SetCookie(w, &http.Cookie{
Name: anubis.CookieName,
Value: tokenString,
Expires: time.Now().Add(24 * 7 * time.Hour),
SameSite: http.SameSiteLaxMode,
Path: "/",
Name: s.opts.CookieName,
Value: tokenString,
Expires: time.Now().Add(24 * 7 * time.Hour),
Domain: s.opts.CookieDomain,
Partitioned: s.opts.CookiePartitioned,
SameSite: http.SameSiteLaxMode,
})

challengesValidated.Inc()
Expand Down
99 changes: 88 additions & 11 deletions lib/anubis_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,39 +1,116 @@
package lib

import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"testing"

"github.com/TecharoHQ/anubis"
"github.com/TecharoHQ/anubis/internal"
"github.com/TecharoHQ/anubis/lib/policy"
)

func spawnAnubis(t *testing.T, h http.Handler) string {
func loadPolicies(t *testing.T, fname string) *policy.ParsedConfig {
t.Helper()

policy, err := LoadPoliciesOrDefault("", anubis.DefaultDifficulty)
if err != nil {
t.Fatal(err)
}

s, err := New(Options{
Next: h,
Policy: policy,
ServeRobotsTXT: true,
})
return policy
}

func spawnAnubis(t *testing.T, opts Options) *Server {
t.Helper()

s, err := New(opts)
if err != nil {
t.Fatalf("can't construct libanubis.Server: %v", err)
}

ts := httptest.NewServer(s)
t.Log(ts.URL)
return s
}

func TestCookieSettings(t *testing.T) {
pol := loadPolicies(t, "")
pol.DefaultDifficulty = 0

srv := spawnAnubis(t, Options{
Next: http.NewServeMux(),
Policy: pol,

t.Cleanup(func() {
ts.Close()
CookieDomain: "local.cetacean.club",
CookiePartitioned: true,
CookieName: t.Name(),
})

return ts.URL
ts := httptest.NewServer(internal.DefaultXRealIP("127.0.0.1", srv))
defer ts.Close()

cli := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}

resp, err := cli.Post(ts.URL+"/.within.website/x/cmd/anubis/api/make-challenge", "", nil)
if err != nil {
t.Fatalf("can't request challenge: %v", err)
}
defer resp.Body.Close()

var chall = struct {
Challenge string `json:"challenge"`
}{}
if err := json.NewDecoder(resp.Body).Decode(&chall); err != nil {
t.Fatalf("can't read challenge response body: %v", err)
}

nonce := 0
elapsedTime := 420
redir := "/"
calcString := fmt.Sprintf("%s%d", chall.Challenge, nonce)
calculated := internal.SHA256sum(calcString)

req, err := http.NewRequest(http.MethodGet, ts.URL+"/.within.website/x/cmd/anubis/api/pass-challenge", nil)
if err != nil {
t.Fatalf("can't make request: %v", err)
}

q := req.URL.Query()
q.Set("response", calculated)
q.Set("nonce", fmt.Sprint(nonce))
q.Set("redir", redir)
q.Set("elapsedTime", fmt.Sprint(elapsedTime))
req.URL.RawQuery = q.Encode()

resp, err = cli.Do(req)
if err != nil {
t.Fatalf("can't do challenge passing")
}

if resp.StatusCode != http.StatusFound {
t.Errorf("wanted %d, got: %d", http.StatusFound, resp.StatusCode)
}

found := false
for _, cookie := range resp.Cookies() {
t.Logf("%#v", cookie)
if cookie.Name == t.Name() {
found = true
}

if found && cookie.Domain != "local.cetacean.club" {
t.Errorf("cookie domain is wrong, wanted local.cetacean.club, got: %s", cookie.Domain)
Comment thread
Xe marked this conversation as resolved.
}
}

if !found {
t.Errorf("Cookie %q not found", t.Name())
}
}

func TestCheckDefaultDifficultyMatchesPolicy(t *testing.T) {
Expand Down
Loading
Loading