FIRES Integration - Moderation Recommendations Feed and General Retractions by sgrigson · Pull Request #71 · eigenmagic/fediblockhole

sgrigson · 2026-03-30T18:05:19Z

This PR intends to bring the FIRES protocol support into Fediblockhole.

It is hoped this will provide many benefits, primary among them is a sort of state management that hasn't existed in Fediblockhole before, where you can check for new updates since your last update and identify retractions as specific entity items.

Everything gets converted (for now) into Mastodon blocklist operations, so drop/reject maps to 'suspend' and 'filter' maps to 'silence'.

The FIRES Project is definitely worth checking out. If Fediblockhole supports RapidBlock, which I'm not even sure is used anymore, it should definitely support FIRES.

There's a production server here at https://fires.1sland.social

You can freely test against the datasets there, or use them in your own projects.

You should pass an 'Accept' header of application/json or the LD-json type to get back JSON data rather than HTML, if you're interested in querying the endpoints directly.

In addition, this builds on the override_private_comment I'd added previously to allow for retractions when something Fediblockhole has added, identified by the override_private_comment, is removed from a list. That's the less safe version of retractions.

You should check out the README in the PR for more notes on how this is intended to work.

ThisIsMissEm · 2026-03-30T19:33:33Z

etc/sample.fediblockhole.conf.toml

+# Optional: max_severity to cap the highest severity applied.
+blocklist_fires_sources = [
+  # { server = 'https://fires.example.com' },  # all datasets on this server
+  # { server = 'https://fires.example.com', datasets = ['dataset-uuid-1', 'dataset-uuid-2'] },


Datasets cannot be addressed via UUID, only the absolute IRI for the dataset. You can go:

GET /.well-known/nodeinfo -> /nodeinfo/2.1 -> metadata.fires.datasets

GET metadata.fires.datasets URI as application/ld+json

And then get the individual datasets that way. This is currently I think best documented through the Conformance Test Suite that's almost ready: fedimod/fires#237 (it works, we're just needing to add a few more things)

The protocol for FediMod FIRES does not actually define any URL structure, instead it's "follow your nose" approach: all resources link to other resources.

Yeah, that's cumbersome and I'm going to do more proper discovery of labels and other things.

I've pushed up some changes, but the .well-known/nodeinfo discovery is probably the way to go at the top level. Thanks.

In the future there'll be a labelsets key in metadata.fires from well-known, and labels will be deprecated (it'll refer to the first labelset create on the server)

…scovery

ThisIsMissEm

Left a heap of comments. Some of them explain the rationale, and are things I haven't yet necessarily clearly stated in the documentation. (pull requests are really welcome! It's a huge amount of work for one person)

ThisIsMissEm · 2026-03-30T19:34:37Z

etc/sample.fediblockhole.conf.toml

+#   1. Server-wide: fetch all datasets from a FIRES server
+#      { server = 'https://fires.example.com' }
+#   2. Cherry-pick: fetch specific datasets from a server by UUID
+#      { server = 'https://fires.example.com', datasets = ['uuid-1', 'uuid-2'] }


Suggested change

# { server = 'https://fires.example.com', datasets = ['uuid-1', 'uuid-2'] }

# { datasets = ['https://fires.example/datasets/uuid-1', 'https://fires.example/datasets/uuid-1'] }

ThisIsMissEm · 2026-03-30T19:34:53Z

etc/sample.fediblockhole.conf.toml

+#   2. Cherry-pick: fetch specific datasets from a server by UUID
+#      { server = 'https://fires.example.com', datasets = ['uuid-1', 'uuid-2'] }
+#   3. Direct URL: paste a dataset URL directly
+#      { url = 'https://fires.example.com/datasets/uuid-1' }


Suggested change

# { url = 'https://fires.example.com/datasets/uuid-1' }

# { dataset = 'https://fires.example.com/datasets/uuid-1' }

ThisIsMissEm · 2026-03-30T19:43:27Z

src/fediblockhole/__init__.py

+    fires_allowlists = []
+    fires_retractions = set()  # domains retracted by trusted FIRES sources
+    if not conf.no_fetch_fires:
+        fires_blocks, fires_allows, fires_retractions = fetch_from_fires(


I generally recommend doing it as a "fetch changes from this dataset", and then apply those changes in order, the changes endpoint is sorted by insertion time (internally each record is tracked with a UUID v7, which is time-ordered).

How you apply those changes is up to your software, but if the entries are (oldest to newest):

recommendation https://a.example recommendedPolicy=drop recommendation https://a.example recommendedPolicy=filter, recommendedFilters=reject-reports recommendation https://a.example recommendedPolicy=accept

Then the final result would be a single rule for https://a.example that is the policy of "accept", with no filters applied.

That is you apply the records in order, and they are not merges but overwrites

ThisIsMissEm · 2026-03-30T19:44:12Z

src/fediblockhole/__init__.py

    return blocklists


+def _parse_dataset_url(url: str) -> tuple:


dataset URLs are not parseable. The reference server just uses this format, but other implementations may not.

To determine if something is or is not a FIRES server, make the request through nodeinfo to discover that information. Then follow your nose from there.

ThisIsMissEm · 2026-03-30T19:46:31Z

src/fediblockhole/__init__.py

+                    for ds in datasets:
+                        ds_id_url = ds.get("id", "")
+                        if ds_id_url:
+                            ds_id = ds_id_url.rstrip("/").split("/")[-1]
+                            fetch_list.append((server_url, ds_id))


Don't parse the URLs, these are the same as id properties in ActivityPub, they are where the object lives, the structure of the URL does not imply any information.

ThisIsMissEm · 2026-03-30T20:26:48Z

README.md

+FIRES recommendations include labels from the
+[IFTAS shared vocabulary](https://about.iftas.org/library/shared-vocabulary-labels/)
+(e.g., "Hate Speech", "CSAM", "Spam"). These are mapped to the `public_comment`
+field on domain blocks, so instance admins can see why a domain was recommended
+for blocking.


This is just one known label vocabulary, based on the Digital Trust & Safety Partnerships' labels which they released as CC-BY license. I know for instance that garden fence has it's own label vocabulary. Others may exist in the future too.

ThisIsMissEm · 2026-03-30T20:28:03Z

README.md

+
+This means an `accept` from a FIRES dataset acts as an override, the same as
+adding a domain to a CSV allowlist. It does not call any instance API to
+explicitly allow the domain — it simply prevents it from being blocked.


This is correct, if you're not doing federation policies and just the binary approach that Mastodon uses of domain blocks OR domain allows.

ThisIsMissEm · 2026-03-30T20:28:59Z

README.md

+
+With `ignore_accept` enabled, `accept` recommendations are silently skipped.
+Block recommendations (`drop`, `reject`, `filter`) and retractions still work
+normally.


This is incorrect, as it is plausible and possible for a Recommendation of drop to become a Recommendation of accept without an intermediary Retraction

ThisIsMissEm · 2026-03-30T20:29:29Z

README.md

+Block recommendations (`drop`, `reject`, `filter`) and retractions still work
+normally.
+
+### Retractions: removing blocks that are no longer recommended


Suggested change

### Retractions: removing blocks that are no longer recommended

### Retractions: removing data that is no longer recommended or advised

ThisIsMissEm · 2026-03-30T20:33:59Z

README.md

+This is the FIRES-native approach. When a trusted FIRES source explicitly
+retracts a domain, the block is removed from your instance — **regardless of
+who originally added it** — as long as no other source in your merged list still
+recommends blocking it.


You would almost certainly want to do bookkeeping to check "did this dataset add this domain as a recommendation/advisory?" as to be able to know if the retraction is valid. Otherwise a retraction from one dataset may override a recommendation from another.

If there is a case of one dataset has the latest state for an entity as retraction or tombstone and another dataset has a advisory or recommendation, you have a few ways of merging that: manual merging, least permissive (i.e., most sever policy applies) or most permissive (least sever policy applies).

You could also do automatic merging and fall back to requesting an operator resolve a merge conflict were an appropriate final policy cannot be determined.

ThisIsMissEm · 2026-03-31T20:41:57Z

src/fediblockhole/__init__.py

+    Optional per-source keys:
+      max_severity   -- cap the highest severity (default: 'suspend')
+      ignore_accept  -- skip 'accept' policy entries (default: false)
+      retractions    -- honor retractions from this source (default: false)


I'd argue this should probably be true but with user interaction

jpwarren · 2026-03-31T22:07:48Z

Thanks very much for making this!

I've set the PR to draft while people are iterating on the code. Once it's stable, we can set it back to ready for merge.

I'll try to find some time in the next couple of days to review the changes and provide any guidance that might be useful on overall architecture or style things if I see any. I don't want to leave you hanging.

ThisIsMissEm

Another round of comments!

ThisIsMissEm · 2026-03-31T20:42:33Z

src/fediblockhole/__init__.py

+        if source_idx > 0:
+            time.sleep(2)


What's this for?

To prevent hitting 429 errors, hopefully. I suppose I could just rely on exponential backoff.

ThisIsMissEm · 2026-03-31T20:44:00Z

src/fediblockhole/__init__.py

        max_severity = source.get("max_severity", "suspend")
        ignore_accept = source.get("ignore_accept", False)
        honor_retractions = source.get("retractions", False)
+        language = source.get("language", "en")


Here's the full list of locales the reference server uses, but really it's just any BCP-47 language tag: https://github.com/fedimod/fires/blob/main/components/fires-server/config/locales.ts (I use this limited list to make the UI approachable)

So this should probably be en-US as that's the default locale: https://github.com/fedimod/fires/blob/main/components/fires-server/start/env.ts#L75

ThisIsMissEm · 2026-03-31T20:47:59Z

src/fediblockhole/__init__.py


+            # Check if this block was added by one of the datasets
+            # that is now retracting it
+            private_comment = getattr(serverblock, 'private_comment', '') or ''


One day the Mastodon team will add like a correlation_id or something to domain blocks and domain allows to allow adding that metadata. Or just an arbitrary metadata json blob that is completely pass-through. I just won't be implementing it for the foreseeable future.

ThisIsMissEm · 2026-03-31T20:48:58Z

src/fediblockhole/fires.py

-    def __init__(self, base_url: str):
-        self.base_url = base_url.rstrip("/")
+    def __init__(self, dataset_url: str):
+        self.dataset_url = dataset_url.rstrip("/")


I'd trust the user's input as verbatim; whilst the reference server doesn't care about the trailing slash, other implementations might.

ThisIsMissEm · 2026-03-31T20:49:38Z

src/fediblockhole/fires.py

+            if response.status_code == 429 and attempt < retries - 1:
+                wait = (attempt + 1) * 5
+                log.warning(f"FIRES: rate limited on {url}, waiting {wait}s (attempt {attempt + 1}/{retries})")
+                time.sleep(wait)
+                continue


ha, you met my rate limiter? What threshold triggered it?

Fwiw, I've just added the documentation for the rate limit headers that are actually present, so you can follow those: adonisjs/v7-docs#30

ThisIsMissEm · 2026-03-31T22:35:46Z

src/fediblockhole/fires.py

+        else:
+            # Try extracting a slug from a URL as fallback
+            slug = label_ref.rstrip("/").split("/")[-1]
+            names.append(slug)


You shouldn't need this, labels must have a name or nameMap

ThisIsMissEm · 2026-03-31T22:38:20Z

src/fediblockhole/fires.py

+    comment = (comment or "").strip()
+    if label_text and comment:
+        return f"{label_text} — {comment}"
+    return label_text or comment


There doesn't seem to be a max-length in mastodon for domain block comments, not sure about other software. It wouldn't be unreasonable for cap this at like 1k or 2k characters, or a max-graphemes count (if UTF-8)

ThisIsMissEm · 2026-03-31T22:41:10Z

src/fediblockhole/fires.py

+
+            # No policy means informational only — skip it
+            if not policy:
+                continue


Might be worth adding a log line here, the validator in the reference server does actually currently enforce all changes have a recommendedPolicy: https://github.com/fedimod/fires/blob/main/components/fires-server/app/validators/admin/dataset_change.ts#L80

I would keep this logic, but just log that you encountered a recommendation without a recommended policy.

ThisIsMissEm · 2026-03-31T22:42:19Z

src/fediblockhole/fires.py

                    domain=domain,
                    severity=severity,
                    public_comment=public_comment,
+                    private_comment=f"FIRES:{dataset_url}",


you may wanna add a sha256 hash of the change record's id too, which should be enough to deduplicate when tombstones are a thing.

ThisIsMissEm · 2026-03-31T22:43:47Z

src/fediblockhole/fires.py

-        log.info(f"FIRES: incremental update for dataset {dataset_id}")
-        snapshot = client.get_snapshot(dataset_id)
+        log.info(f"FIRES: incremental update for {dataset_url}")
+        snapshot = client.get_snapshot()


this is really get_changes here, I can see why you might merge those methods, but I'd probably keep them separate.

sgrigson added 2 commits March 28, 2026 23:50

FIRES Integration And Retractions

c92aff1

change label behavior into comments

4ab7acc

sgrigson changed the title ~~FIRES Integration - Moderation Recommendations Feed~~ FIRES Integration - Moderation Recommendations Feed and General Retractions Mar 30, 2026

ThisIsMissEm reviewed Mar 30, 2026

View reviewed changes

sgrigson added 2 commits March 30, 2026 14:40

removing a lot of cumbersome url construction in favor of true autodi…

d1a6de8

…scovery

handle 429 retries/backoff

61055dd

sgrigson requested a review from ThisIsMissEm March 30, 2026 19:50

ThisIsMissEm suggested changes Mar 30, 2026

View reviewed changes

sgrigson added 4 commits March 30, 2026 17:56

cleanup - em's feedback

e6f03fb

no need to support old format

53b4339

clarify what ignore_accept does now

b5efa02

fixed example config in docstring

79edf03

sgrigson requested a review from ThisIsMissEm March 31, 2026 13:46

language handling

a78a491

ThisIsMissEm reviewed Mar 31, 2026

View reviewed changes

jpwarren marked this pull request as draft March 31, 2026 22:02

jpwarren added enhancement New feature or request kudos! You made the world a bit better. Thanks! labels Mar 31, 2026

jpwarren added this to the v0.5.0 milestone Mar 31, 2026

ThisIsMissEm reviewed Mar 31, 2026

View reviewed changes

	# { server = 'https://fires.example.com', datasets = ['uuid-1', 'uuid-2'] }
	# { datasets = ['https://fires.example/datasets/uuid-1', 'https://fires.example/datasets/uuid-1'] }

	# { url = 'https://fires.example.com/datasets/uuid-1' }
	# { dataset = 'https://fires.example.com/datasets/uuid-1' }

	### Retractions: removing blocks that are no longer recommended
	### Retractions: removing data that is no longer recommended or advised

Conversation

sgrigson commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sgrigson Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ThisIsMissEm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jpwarren commented Mar 31, 2026

Uh oh!

ThisIsMissEm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sgrigson commented Mar 30, 2026 •

edited

Loading

sgrigson Mar 30, 2026 •

edited

Loading