Skip to content

feat(mikrotik): structured-output parsing base + validation fix (7/10) #370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
WilhelmZA wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(mikrotik): structured-output parsing base + validation fix (7/10) #370

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
218 changes: 209 additions & 9 deletions docs/pages/configuration/config/structured-output.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,41 @@
import { Callout } from "nextra/components";

## Structured

Devices that support responding to a query with structured or easily parsable data can have their response data placed into an easier to read table (or JSON, when using the REST API). Currently, the following platforms have structured data supported in hyperglass:

- Arista EOS
- FRRouting
- Huawei VRP
- Juniper Junos
- Mikrotik RouterOS/SwitchOS

When structured output is available, hyperglass checks the RPKI state of each BGP prefix returned using one of two methods:

1. From the router's perspective
2. From the perspective of [Cloudflare's RPKI Service](https://rpki.cloudflare.com/)
1. **From the router's perspective** - Uses the RPKI validation state as determined by the router itself
2. **From an external RPKI validation service** - Queries an external service to determine RPKI state independently

For external validation, hyperglass supports two backends:
- **Cloudflare**: Uses [Cloudflare's public RPKI service](https://rpki.cloudflare.com/) via GraphQL API
- **Routinator**: Connects to your own [Routinator](https://github.com/NLnetLabs/routinator) RPKI validator instance

Additionally, hyperglass provides the ability to control which BGP communities are shown to the end user.

| Parameter | Type | Default Value | Description |
| :----------------------------- | :-------------- | :------------ | :---------------------------------------------------------------------------------------------------------------------------- |
| `structured.rpki.mode` | String | router | Use `router` to use the router's view of the RPKI state (1 above), or `external` to use Cloudflare's view (2 above). |
| `structured.communities.mode` | String | deny | Use `deny` to deny any communities listed in `structured.communities.items`, or `permit` to _only_ permit communities listed. |
| `structured.communities.items` | List of Strings | | List of communities to match. |
For devices with structured traceroute support (FRRouting, Huawei VRP, and MikroTik RouterOS), hyperglass can enhance the output with IP enrichment data including ASN information, organization names, country codes, and IXP detection using real-time lookups from BGP.tools.

| Parameter | Type | Default Value | Description |
| :-------------------------------- | :-------------- | :------------ | :------------------------------------------------------------------------------------------------------------------------------------- |
| `structured.rpki.mode` | String | router | Use `router` to use the router's view of the RPKI state, or `external` to use an external validation service. |
| `structured.rpki.backend` | String | cloudflare | When using `external` mode, choose `cloudflare` or `routinator` as the validation backend. |
| `structured.rpki.rpki_server_url` | String | | When using `routinator` backend, specify the base URL of your Routinator server (e.g., `http://rpki.example.com:3323`). |
| `structured.communities.mode` | String | deny | Use `deny` to deny any communities listed, `permit` to _only_ permit communities listed, or `name` to append friendly names. |
| `structured.communities.items` | List of Strings | | List of communities to match (used by `deny` and `permit` modes). |
| `structured.communities.names` | Dict | | Dictionary mapping BGP community codes to friendly names (used by `name` mode). |
| `structured.ip_enrichment.cache_timeout` | Integer | 604800 | **DEPRECATED:** No longer used with real-time BGP.tools lookups. Kept for backward compatibility only. |
| `structured.ip_enrichment.enrich_traceroute`| Boolean | false | When `structured:` is present, enable IP enrichment of traceroute hops (ASN, org, IXP). This must be true for enrichment to run. |
| `structured.ip_enrichment.enrich_bgproute` | Boolean | false | When `structured:` is present, enable early ASN/org enrichment for BGP routes. When true, enrichment happens immediately on query response. When false, enrichment is lazy-loaded when viewing AS Path. |
| `structured.enable_for_traceroute`| Boolean | (when structured present) true | When `structured:` is present this controls whether the structured traceroute table output is shown. Set to false to force raw router output. |
| `structured.enable_for_bgp_route`| Boolean | (when structured present) true | When `structured:` is present this controls whether the structured BGP route table output is shown. Set to false to force raw router output. |

### RPKI Examples

Expand All @@ -28,14 +47,31 @@ structured:
mode: router
```

#### Show RPKI State from a Public/External Perspective
#### Show RPKI State from Cloudflare's Public Service

```yaml filename="config.yaml" copy {2}
```yaml filename="config.yaml" copy {2-4}
structured:
rpki:
mode: external
backend: cloudflare
```

#### Show RPKI State from a Custom Routinator Server

```yaml filename="config.yaml" copy {2-5}
structured:
rpki:
mode: external
backend: routinator
rpki_server_url: "http://rpki.example.com:8080"
```

<Callout type="info" emoji="ℹ️">
**Routinator URL Format**

The `rpki_server_url` should be the base URL of your Routinator HTTP web API endpoint. This is typically different from the RTR port (3323). The URL should not include the `/validity` path as hyperglass will append this automatically.
</Callout>

### Community Filtering Examples

#### Deny Listed Communities by Regex pattern
Expand All @@ -59,3 +95,167 @@ structured:
- "^65000:.*$" # permit any communities starting with 65000, but no others.
- "1234:1" # permit only the 1234:1 community.
```

#### Append Friendly Names to Communities

```yaml filename="config.yaml" {2-10}
structured:
communities:
mode: name
names:
"65000:1000": "Upstream Any"
"65000:1001": "Upstream A (all locations)"
"65000:1101": "Upstream A Location 1"
"65000:1201": "Upstream A Location 2"
"65000:1002": "Upstream B (all locations)"
"65000:1102": "Upstream B Location 1"
"65000:2000": "IXP Any"
```

### IP Enrichment Examples

<Callout type="info" emoji="ℹ️">
**IP Enrichment Requirements**

IP enrichment is currently supported for traceroute outputs on supported platforms.

The system uses real-time lookups from BGP.tools for maximum accuracy and reliability.
</Callout>

#### Enable IP Enrichment for Traceroute

```yaml filename="config.yaml" copy {2-4}
structured:
# Ensure `structured:` exists to enable structured output. By default the
# structured table output is enabled when this block is present. To disable
# the structured traceroute table, set `structured.enable_for_traceroute: false`.
ip_enrichment:
enrich_traceroute: true
```

#### Enable IP Enrichment

```yaml filename="config.yaml" copy {2-4}
structured:
ip_enrichment:
enrich_traceroute: true
# cache_timeout is deprecated - real-time lookups are used
```

#### Enable IP Enrichment for Traceroute

```yaml filename="config.yaml" copy {2-3}
structured:
ip_enrichment:
enrich_traceroute: true
```

<Callout type="warning" emoji="⚠️">
**Performance Considerations**

- Initial cache loading may take 30-60 seconds on first startup
- Data is cached locally using pickle format for ultra-fast subsequent loads
- Cache files are stored in `/etc/hyperglass/ip_enrichment/`
- Minimum cache timeout is 24 hours (86400 seconds) to prevent excessive API usage
</Callout>

### Structured Traceroute Configuration

<Callout type="info" emoji="ℹ️">
**Structured Traceroute Support**

Structured traceroute with rich metadata is available for:
- **FRRouting**: Parses Unix-style traceroute output with load balancing and multi-path support
- **Huawei VRP**: Parses Unix-style traceroute output
- **MikroTik RouterOS/SwitchOS**: Parses multi-table format with statistics

When IP enrichment is enabled, traceroute hops are enhanced with ASN numbers, organization names, country codes, prefixes, and IXP detection.
</Callout>

#### Complete Structured Traceroute Setup

```yaml filename="config.yaml" copy {2-12}
structured:
rpki:
mode: external
backend: routinator
rpki_server_url: "https://rpki.example.com"
communities:
mode: name
names:
"65000:1000": "Transit Routes"
"65000:2000": "Peer Routes"
ip_enrichment:
enrich_traceroute: true
# cache_timeout is deprecated - real-time lookups are used
```

#### Structured Traceroute with Cloudflare RPKI

```yaml filename="config.yaml" copy {2-9}
structured:
rpki:
mode: external
backend: cloudflare
ip_enrichment:
enrich_traceroute: true
```

#### Minimal Structured Traceroute (No IP Enrichment)

```yaml filename="config.yaml" copy {2-4}
structured:
ip_enrichment:
enrich_traceroute: false # Traceroute will show basic hop info without ASN/org data
enrich_bgproute: true # BGP routes will include ASN/org enrichment on query response
```

<Callout type="warning" emoji="⚠️">
**IP Enrichment Dependency**

Without IP enrichment enabled:
- Traceroute hops will only show IP addresses and RTT values
- BGP next-hop IPs will not show ASN/organization tooltips
- No ASN organization names in copied BGP route text
- AS path visualization will be limited or unavailable
- IXP detection will not function

For the full structured traceroute and BGP route experience with rich metadata, enable the appropriate enrichment settings.
</Callout>

### BGP Route Enrichment

BGP route enrichment provides two modes for ASN and organization lookups:

#### Early Enrichment (On-Query)
When `enrich_bgproute: true`, enrichment happens immediately during query execution:
- ✅ ASN names appear instantly in AS path entries
- ✅ Next-hop IP tooltips show organization information
- ✅ Copied text includes enriched data: `37468 (Organization) → 12956 (Organization)`
- ⚠️ Adds a small delay to query execution for bulk lookups

```yaml filename="config.yaml" copy {4}
structured:
ip_enrichment:
enrich_bgproute: true
```

#### Lazy Enrichment (On-Demand)
When `enrich_bgproute: false` (default), enrichment happens only when needed:
- ✅ Faster initial query response
- ✅ Organization names load when clicking "View AS Path" button or hovering
- ✅ Reduces unnecessary lookups for partial queries
- ⚠️ Organization names not included in copied text by default

```yaml filename="config.yaml" copy {4}
structured:
ip_enrichment:
enrich_bgproute: false
```

#### Next-Hop Enrichment

Regardless of enrichment mode, next-hop IPs for BGP routes are always enriched with:
- Peer ASN (if available)
- Peer organization name (if available)
- Hovering over a next-hop IP shows: `192.0.2.1 (ASN Organization)`
9 changes: 8 additions & 1 deletion hyperglass/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,14 @@

TARGET_JUNIPER_ASPATH = ("juniper", "juniper_junos")

SUPPORTED_STRUCTURED_OUTPUT = ("frr", "juniper", "arista_eos")
SUPPORTED_STRUCTURED_OUTPUT = (
"frr",
"juniper",
"arista_eos",
"huawei",
"mikrotik_routeros",
"mikrotik_switchos",
)

CONFIG_EXTENSIONS = ("py", "yaml", "yml", "json", "toml")

Expand Down
8 changes: 4 additions & 4 deletions hyperglass/models/config/devices.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,10 +247,10 @@ def validate_structured_output(cls, value: bool, info: ValidationInfo) -> bool:
if value is True:
if info.data.get("platform") not in SUPPORTED_STRUCTURED_OUTPUT:
raise ConfigError(
"The 'structured_output' field is set to 'true' on device '{}' with "
+ "platform '{}', which does not support structured output",
info.data.get("name"),
info.data.get("platform"),
"The 'structured_output' field is set to 'true' on device '{d}' with "
"platform '{p}', which does not support structured output",
d=info.data.get("name"),
p=info.data.get("platform"),
)
return value
if value is None and info.data.get("platform") in SUPPORTED_STRUCTURED_OUTPUT:
Expand Down
Loading