Skip to content

Commit 71b8b80

Browse files
sebixsebix
committed
doc: abuse contacts: add models, CRM tools, graphs
1 parent b614def commit 71b8b80

1 file changed

Lines changed: 180 additions & 25 deletions

File tree

‎docs/user/abuse-contacts.md‎

Lines changed: 180 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,196 @@
11
<!-- comment
2-
SPDX-FileCopyrightText: 2015-2023 Sebastian Wagner, Filip Pokorný
2+
SPDX-FileCopyrightText: 2021 CERT.at GmbH, 2023 Filip Pokorný, 2026 Institute for Common Good Technology
33
SPDX-License-Identifier: AGPL-3.0-or-later
44
-->
55

6-
76
# Abuse-contact look-ups
87

9-
The right decision whom to contact about a specific incident is vital to get the incident resolved as quick as possible. Different types of events may required different abuse-contact to be selected. For example, issues about a device, e.g. a vulnerability in the operating system or an application, is better sent to the hoster which can inform the server administrator. For website-related issues, like defacements or phishing, the domain owner (maintaining the content of the website) could be the better and more direct contact. Additionally, different CERT's have different approaches and different contact databases. Multiple information sources have different information, and some sources are more accurate than others. IntelMQ can query multiple sources of abuse-contacts and combine them. Internal databases, like a Constituency Portal provide high-quality and first-hand contact information. The RIPE document [Sources of Abuse Contact Information for Abuse Handlers](https://www.ripe.net/publications/docs/ripe-658) contains a good summary of the complex of themes.
8+
Choosing the right notification contact for a security incident or vulnerability is essential to getting it resolved quickly.
9+
The correct contact depends on the incident type: a device vulnerability is best reported to the hosting provider (who can reach the server administrator), while a defacement or phishing site is better directed to the domain owner responsible for the content.
1010

11-
## Sources for abuse-contacts
11+
Different organisations also have different contact databases and workflows.
12+
A SOC typically manages contacts for its own organisation and subsidiaries, while a national CSIRT covers all networks within a country.
13+
IntelMQ supports both scenarios: it offers multiple tool suits and can query multiple abuse-contact sources and combine the results.
14+
15+
Management suites for constituencies, contacts and network objects provide the highest-quality contact data because network owners maintain it themselves.
16+
Public sources like RIPE are broader and less precise, but provide a very good base and are also suitable for finding related infrastructure in the first place.
17+
18+
The RIPE document [Sources of Abuse Contact Information for Abuse Handlers](https://www.ripe.net/publications/docs/ripe-658) gives a thorough overview of the available sources and their trade-offs.
1219

13-
All these bots add the queried contacts to the IntelMQ events in the field `source.abuse_contact` if not state otherwise in the documentation.
20+
## Suites for contact management and CRM
21+
22+
There are two established integrated solutions for managing contacts and networks.
23+
Both of them are made for and integrated in IntelMQ.
24+
They share also parts of their database models and the RIPE import tooling.
25+
26+
### CERTBund
27+
28+
The CERTBund suite was originally created for [CERTBund](https://www.cert-bund.de/) and provides a dual level contact management for automatic and manual contacts.
29+
The processing is highly configurable by user-defined rules and formatting scripts.
30+
31+
- [Documentation](https://intevation.github.io/intelmq-mailgen/)
32+
- [Installation](../admin/installation/combined.md)
33+
- The [combined installation documentation](../admin/installation/combined.md) includes the installation of this suite
34+
35+
It consists of these components:
36+
37+
- [`intelmq-certbund-contact`](https://github.com/Intevation/intelmq-certbund-contact/): two IntelMQ expert bots (Contact-Expert and Rule-Expert) that enrich events with contact information from the contact database.
38+
Based on user-defined rules they determine the recipient, email template, and attachment format.
39+
- RIPE import scripts keep contact data up to date automatically.
40+
- [`intelmq-fody`](https://github.com/Intevation/intelmq-fody) & [`intelmq-fody-backend`](https://github.com/Intevation/intelmq-fody-backend): web interface for reading and editing contacts, querying sent notification tickets, and browsing the IntelMQ event database.
41+
- [`intelmq-mailgen`](https://github.com/Intevation/intelmq-mailgen): tool that reads the notification directives written by the CERTBund experts and sends grouped event emails to the determined contacts.
42+
Supports OpenPGP encryption, formatting is configured by scripts and templates.
43+
- `contactdb`: The PostgreSQL database that covers Autonomous Systems, network ranges (CIDR), single IP addresses, and domains.
44+
45+
46+
```mermaid
47+
---
48+
config:
49+
layout: dagre
50+
---
51+
graph TD
52+
subgraph intelmq["IntelMQ Core"]
53+
collectors["Collectors"] --> parsers["Parsers"]
54+
parsers --> cb_contact["CERTBundContact-Expert"]
55+
cb_contact --> cb_rules["CERTBundRules-Expert"]
56+
cb_rules --> output["SQL Output"]
57+
end
58+
subgraph certbundcontact["IntelMQ CERTBund Contact"]
59+
rules[("Notification Rules")] --> cb_rules
60+
ripe[("RIPE")] -- ripe_import --> contactdb[("ContactDB")]
61+
manual[("manual contacts")] --> contactdb
62+
contactdb --> cb_contact
63+
fody["Fody (web UI)"] --> contactdb
64+
end
65+
66+
subgraph mailgeng["IntelMQ Mailgen"]
67+
output -- write events & directives --> eventdb[("EventDB (intelmq-events)")]
68+
69+
templates[("Templates")] --> mailgen["intelmq-mailgen"]
70+
formats[("Notification Formats")] --> mailgen
71+
eventdb -- read directives --> mailgen
72+
mailgen -- send grouped emails --> recipients["Constituents"]
73+
end
74+
75+
```
76+
77+
78+
**Data model:**
79+
80+
```mermaid
81+
erDiagram
82+
organisation ||--o{ contact : has
83+
organisation ||--o{ ASNs : owns
84+
organisation ||--o{ CIDRs : owns
85+
organisation ||--o{ FQDNs : owns
86+
contact }o--o{ Tags : "tagged with"
87+
organisation }o--o{ Tags : "tagged with"
88+
ASNs }o--o{ Tags : "tagged with"
89+
FQDNs }o--o{ Tags : "tagged with"
90+
CIDRs }o--o{ Tags : "tagged with"
91+
92+
events ||--o{ directives : "triggers (on INSERT)"
93+
directives }o--o| sent : "fulfilled by"
94+
```
95+
96+
Contrary to tuency, there is no hierarchy of organisations.
97+
98+
### Tuency
99+
100+
[Tuency](https://gitlab.com/intevation/tuency/tuency) is a constituency portal that lets organisations self-administer their network objects and notification preferences.
101+
It is maintained by [CERT.at](https://cert.at).
102+
103+
- [Source Code](https://gitlab.com/intevation/tuency/tuency)
104+
- [Documentation](https://tuency.cert.at/docs/)
105+
106+
**Components**
107+
108+
- **Tuency portal**: Web application where constituency members manage their organisations, network objects, and notification rules. Organisations can claim network objects (ASes, IP ranges, domains) themselves, all changes require approval by a tenant admin to prevent hostile takeovers and incorrect data.
109+
- [**Keycloak**](https://www.keycloak.org/): Handles authentication for the Tuency portal and enables single sign-on with other tools in the stack.
110+
- **[Request Tracker for Incident Response](https://bestpractical.com/rtir/) (RTIR)**: Ticketing system used to manage incidents and send notification emails to constituents. The `intelmq.bots.outputs.rt.output` bot creates an Incident ticket in RTIR for each event, and optionally a linked Investigation ticket containing the event details that RTIR then delivers to the contact in `source.abuse_contact`.
111+
- [**intelmq extensions**](https://github.com/certat/intelmq-extensions/tree/main/intelmq_extensions/bots):
112+
- Custom harmonization fields in the IntelMQ data format definition that are required by the workflow
113+
- Tools to create tickets RTIR and deliver notifications to constituents
114+
- Tuency Expert: IntelMQ expert bot that queries the Tuency API to retrieve the appropriate abuse contact and notification interval.
115+
116+
117+
```mermaid
118+
---
119+
config:
120+
layout: dagre
121+
---
122+
graph TD
123+
subgraph intelmq["IntelMQ"]
124+
collectors["Collectors"] --> parsers["Parsers"]
125+
parsers --> expert["Tuency-Expert"]
126+
expert -->|"enriched event"| output["RT Output"]
127+
end
128+
rtir -- "sends notification emails" --> constituents["Constituents"]
129+
consti["Constituents"] -- "self-service" --> tuencyport
130+
131+
tuencyport --> keycloak["Keycloak"]
132+
tuencyport --> contactdb
133+
expert -->|"queries"| tuencyport[("Tuency portal")]
134+
output -- "creates tickets" --> rtir["RTIR"]
135+
```
136+
137+
**Data model:**
138+
139+
- **Tenant** (constituency): a sector CSIRT responsible for a set of organisations. Tuency supports multiple tenants.
140+
- **Organisation**: belongs to a tenant and owns network assets, can have sub-organisations to reflect internal hierarchies.
141+
- **Contacts**: assigned roles such as CISO or 24/7 support, tied to a single organisation.
142+
- **Netobjects** (Network objects): Autonomous Systems, IP address ranges (CIDR), and domains owned and administered by an organisation. All claims go through an approval process.
143+
144+
```mermaid
145+
erDiagram
146+
tenant ||--o{ organisation : "responsible for"
147+
organisation ||--o{ organisation : "sub-organisation of"
148+
organisation ||--o{ contact : has
149+
organisation ||--o{ netobject : "owns and administers"
150+
netobject {
151+
string type "AS / IP range / domain"
152+
int asn "e.g. 1234"
153+
cidr network "e.g. 10.0.0.0/8"
154+
string domain "e.g. example.com"
155+
}
156+
contact {
157+
string role "CISO / 24-7 / …"
158+
string email "e.g. abuse@example.com"
159+
string firstname "e.g. Jane"
160+
string lastname "e.g. Doe"
161+
string tel "e.g. +43 1 234 5678"
162+
}
163+
```
164+
165+
The hierarchical model allows three privilege tiers to control access and organise permissions:
166+
- Orgadmin (organisation admins) that manage their own organisation and all of their sub-organisations
167+
- Tenantadmin manage all organisations in their tenant (constituency)
168+
- Portaladmins manage all organisations in all tenants
169+
170+
## Sources for abuse-contacts
14171

15-
## Sources for domain-based abuse-contacts
172+
The following bots can be combined to build custom contact lookup workflows.
173+
A common pattern is to use high-quality data sources as CERTBund and Tuency first, and fall back to generic public sources for any network resources that are not covered by the internal data sources.
16174

17-
These bots are suitable for domain-based abuse-contact look-ups.
175+
Unless noted otherwise, each bot writes its result to `source.abuse_contact`.
18176

19-
- `intelmq.bots.experts.rdap.expert` expert queries private and public RDAP servers for `source.fqdn` and add the contact information to the event as `source.abuse_contact`.
20-
- `intelmq.bots.experts.trusted_introducer_lookup.expert` expert queries a locally cached [Trusted Introducer team directory](https://www.trusted-introducer.org/directory/teams.json) for the TLD or domain (first match) of `source.fqdn`.
177+
### Domain-based
21178

22-
## Sources for IP address-based abuse-contacts
179+
- `intelmq.bots.experts.rdap.expert` queries private and public RDAP servers for `source.fqdn`.
180+
- `intelmq.bots.experts.trusted_introducer_lookup.expert` looks up the TLD or domain (first match) of `source.fqdn` in a locally cached [Trusted Introducer team directory](https://www.trusted-introducer.org/directory/teams.json).
23181

24-
These bots are suitable for IP address and ASN based abuse-contact look-ups.
182+
### IP address / ASN-based
25183

26-
- `intelmq.bots.experts.abusix.expert` expert queries the online Abusix service.
27-
- `intelmq.bots.experts.do_portal.expert` expert queries an instance of the do-portal software (deprecated).
28-
- `intelmq.bots.experts.tuency.expert` expert queries an instance of the **tuency** Constituency Portal for the IP address. The Portal also takes into account any notification rules, which are saved
29-
additionally in the event.
30-
- `intelmq.bots.experts.ripe.expert` expert queries the online RIPE database for IP-Address and AS contacts.
31-
- `intelmq.bots.experts.trusted_introducer_lookup.expert` expert queries a locally
32-
cached [Trusted Introducer team directory](https://www.trusted-introducer.org/directory/teams.json)
33-
for the Autonomous system `source.asn`.
184+
- `intelmq.bots.experts.abusix.expert` queries the online [Abusix](https://abusix.com/) service.
185+
- `intelmq.bots.experts.do_portal.expert` queries an instance of the do-portal software (deprecated).
186+
- `intelmq.bots.experts.tuency.expert` queries a [Tuency](#tuency) constituency portal. Also retrieves notification rules, which are stored in `extra.ttl`.
187+
- `intelmq.bots.experts.ripe.expert` queries the online RIPE database for IP address and ASN contacts.
188+
- `intelmq.bots.experts.trusted_introducer_lookup.expert` looks up `source.asn` in a locally cached [Trusted Introducer team directory](https://www.trusted-introducer.org/directory/teams.json).
34189

35-
## Generic sources for abuse-contacts
190+
### Generic and local sources
36191

37-
- `intelmq.bots.experts.generic_db_lookup.expert` expert for local data sources, like
38-
database tables mapping ASNs to abuse-contact or Country Codes to abuse-contact.
39-
- `intelmq.bots.experts.uwhoisd.expert` expert for fetching whois-data, not extracting
40-
abuse-contact information
192+
- `intelmq.bots.experts.generic_db_lookup.expert` queries a local database table, e.g. a mapping from ASNs or country codes to abuse contacts.
193+
- `intelmq.bots.experts.uwhoisd.expert` fetches raw WHOIS data, does not extract an abuse contact itself.
41194

42195
## Helpful other bots for pre-processing
43196

@@ -58,7 +211,9 @@ These bots are suitable for IP address and ASN based abuse-contact look-ups.
58211

59212
## Combining the lookup approaches
60213

61-
In order to get the best contact, it may be necessary to combine multiple abuse-contact sources. IntelMQ's modularity provides methods to arrange and configure the bots as needed. Among others, the following bots can help in getting the best result:
214+
In order to get the best contact, it may be necessary to combine multiple abuse-contact sources.
215+
IntelMQ's modularity provides methods to arrange and configure the bots as needed.
216+
Among others, the following bots can help in getting the best result:
62217

63218
- `intelmq.bots.experts.filter.expert` Your lookup process may be different for different types of data. E.g. website-related issues may be better addressed at the domain owner and device-related issues may be better addressed to the hosting provider.
64219
- `intelmq.bots.experts.modify.expert` Allows you to set values based on filter and also format values based on the value of other fields.

0 commit comments

Comments
 (0)