blog/demystifying-router-late/

[giscus[bot]]

blog/demystifying-router-late/

What is the purpose of the ROUTER_LATE role, and when should it be used?

https://meshtastic.org/blog/demystifying-router-late/

[shalberd]

@GUVWAF @thebentern @erayd Hello, a small comment

This has the effect of ensuring that ROUTER_LATE will politely 'give way' to other nodes, thereby preserving the normal routing behaviour of the mesh. Other than the obviously higher airtime used, the impact of deploying a ROUTER_LATE node is identical to as if a CLIENT were deployed at that location.

all true, very well-written, thank you.

It is intended to be deployed in sites that do not have the very large coverage footprint typically suited to ROUTER nodes. Sites that are critical for reliable passage of traffic, but known to not be optimally sited for serving the overall mesh, should use ROUTER_LATE."

That is not entirely true ...

The focus of the article should be less about placement of nodes, but rather about rebroadcast priority and early vs normal vs late contention window. That explanation on contention windows and rebroadcoast priority is very well-written in the blog article, thank you.

In our setups, with high mountain nodes widely visible to all sides, think alpine nodes and surrounding great plains, the problem was that ROUTER always rebroadcast and immediately / early grabbed packets from nodes as far away as 80 km. That led to the absurd situation of routers on mountains relaying packets among each other, while not letting clients near to each other and with perfect 0 hop signal quality relay packets to each other.

The green and red nodes are Clients with perfect 0 hop signal quality. The blue nodes were high-altitude, omni-directionally-visible mountain nodes with role Router:

We had to set the mountain nodes to ROUTER_LATE, allowing nodes in a large square-km area to talk to each other directly.
Router literally destroyed our Mesh.

So, the explanation that ROUTER_LATE is only applicable to nodes not having a large coverage footprint is definitely not correct in our experience. In fact, we had to set mountain nodes from ROUTER to ROUTER_LATE to avoid a country-wide round-trip tour of Switzerland on perfectly situated nodes with an extreme footprint to all 4 directions over large distances.

See https://github.com/orgs/meshtastic/discussions/24#discussioncomment-14397405

I discussed the issues we faced with @GUVWAF here:

meshtastic/firmware#6904 (comment)

We then followed-up in our Swiss group:

and https://github.com/orgs/meshtastic/discussions/24#discussioncomment-14094052

If your node's coverage footprint is genuinely excellent, please keep it as ROUTER. ROUTER will preempt all other roles, and is effectively asserting that it is the best path for all traffic within range.

I disagree, especially when a node is placed top-notch with long-distance coverage, ROUTER is hurting the mesh. This gets even worse as more than one Router is added.

The delayed rebroadcast window of ROUTER_LATE is essential for mountain nodes with wide, omni-directional, long-distance coverage.

In lateral valleys not accessible from all 4 directions, Client Role is totally sufficient for forwarding inside a valley. That is true even if the node is on either high side of an e.g. V-shaped valley, but not completely on the crest, making east-west or north-south comms possible, but not all 4 directions.

Please note that ROUTER and REPEATER nodes will cause all rebroadcast roles other than ROUTER, REPEATER, ROUTER_LATE and CLIENT_BASE (for packets to / from favourite nodes only) within their coverage area to cancel their own rebroadcasts - so only deploy ROUTER nodes in locations that are actually properly suited to that behaviour.

Because of this, it should be emphasized that especially when long-distance comms from a router node the the surrounding areas is possible, it actually leads to state / province / country-wide round-trips.

We are even of the opinion that the ultra-aggressive (in terms of timing) ROUTER and REPEATER roles are no longer relevant and hurting a mesh more than they help, but that of course is subjective from our experience.

Sorry, not a small comment after all, but very important, I think. In our experience, the agressive timing, combined with the term "router", is suboptimal, in hindsight.

[erayd]

ROUTER cannot ever prevent two nodes with a good direct path from talking directly to each other. In all cases, if node B can receive node A directly, and node A transmits something, node B will receive it before any other node gets the chance to rebroadcast it. This is because node B will receive it at the same time as other nodes that can also hear A well.

In the case of your red and green nodes, if your trace between them went via the ROUTER nodes first, it went that way because the destination node could not hear the original sender directly. This could be due to a number of factors - terrain obstruction, RF interference, simultaneous transmission, etc. It may also be that there is an asynchronous path there, where the RF setup is such that they can communicate directly in one direction, but not the other.

Looking at mapping for those particular sites, it does appear that there is some terrain in the way between them - perhaps that is the issue in the particular case you have illustrated above?

---

Separately to the issue of obstructions, is the problem of having too many ROUTER nodes. This is absolutely a problem on many meshes, and can cause the silly 'hopping all over the region' type behaviour that your image illustrates between the blue nodes. Having too many ROUTER nodes does not mean the description of when a ROUTER is appropriate is wrong - it simply means you have too many of them, and are ignoring the rest of the answer to that question, which addresses the context. You'll note that I explicitly call out that ROUTER should only be deployed in locations that are actually suited to the rebroadcast-cancelling behaviour. That includes due consideration of what other nodes in the area may be doing (e.g. whether there are already other ROUTER nodes nearby that may be trying to do the same thing).

Do you think an addition to the post that expands on the "...will preempt all other roles, and is effectively asserting that it is the best path for all traffic within range." part to further emphasise consideration of what other ROUTER nodes may already be present would be helpful?

In our setups, with high mountain nodes widely visible to all sides, think alpine nodes and surrounding great plains, the problem was that ROUTER always rebroadcast and immediately / early grabbed packets from nodes as far away as 80 km. That led to the absurd situation of routers on mountains relaying packets among each other, while not letting clients near to each other and with perfect 0 hop signal quality relay packets to each other.

Meshtastic infrastructure nodes must set up with the terrain they are serving in mind. In the kind of situation you describe, having the odd really high ROUTER is helpful, because it minimises the number of hops needed to get a packet between two distant points. However, it is important to keep the number of them to a minimum. Then ROUTER_LATE can be used to fill in the gaps - that is the main reason the ROUTER_LATE role exists. It is supposed to ensure that packets get repeated in places where it is essential that they be repeated, regardless of whether another node has already done so, but without disturbing other possible paths.

If you have a potential path that can go around an obstruction via 2-3 other nodes, or over the top of it via just one, it's obviously desirable to force it to go over the top. A appropriately-sited ROUTER achieves precisely this - it rapidly gets packets up out of the low clutter, prevents unnecessary retransmissions from other nodes within that clutter, and moves the packet a long way. Whether it is a good idea to do this will also depend on what other effects putting a ROUTER node at that site may cause.

In lateral valleys not accessible from all 4 directions, Client Role is totally sufficient for forwarding inside a valley. That is true even if the node is on either high side of an e.g. V-shaped valley, but not completely on the crest, making east-west or north-south comms possible, but not all 4 directions.

This depends on what else the clients can see. A common scenario in such a valley is for a node on the valley floor to send a packet, another node on the valley floor to repeat it, and then the higher node up on the side refuses to rebroadcast, because it saw both the original transmission, and the rebroadcast from the second valley floor node. This can often lead to the packet ending its journey there, and never making it through to the rest of the valley (or in fact elsewhere on the mesh either). Setting that higher node to ROUTER_LATE instead of CLIENT ensures that the packet gets sent from that node on the high side of the valley regardless, but will not interfere with what the CLIENT nodes may do amongst themselves.

We are even of the opinion that the ultra-aggressive (in terms of timing) ROUTER and REPEATER roles are no longer relevant and hurting a mesh more than they help...

I strongly disagree with this (although I do think REPEATER should be killed off, but that is because it's invisible, not because of the timing behaviour). ROUTER serves a very valuable purpose, and there is at present no other way to achieve the same goal of having a node aggressively assert that it is the best path. I think the issue is primarily an education one - that ROUTER should only be used in situations where it genuinely makes sense to do so, with a full understanding of the implications.

There is currently a proposal to have CLIENT nodes ignore rebroadcasts by a ROUTER for the purposes of deciding whether to cancel their own rebroadcast of a packet. This is IMO quite a good thing, as it means that while ROUTER will still preempt in terms of timing, it will not disable CLIENT paths the way it does currently. Will be interesting to see what comes of this, as it will help to reduce the damage that inappropriate role choice can cause.

---

At the end of the day, I am of the opinion that anybody who is deploying infrastructure that preempts the standard CLIENT meshing has a duty of care to ensure that they then make such coverage sufficiently comprehensive, and to fully understand what it is that they are doing. If you put a ROUTER somewhere that will cause packets to no longer reach a particular part of the mesh, then you also have a responsibility to ensure that there is other infrastructure in place to fill that gap.

Because of this, it should be emphasized that especially when long-distance comms from a router node the the surrounding areas is possible, it actually leads to state / province / country-wide round-trips. [...] Sorry, not a small comment after all, but very important, I think.

I agree. Talking about this stuff is helpful - it means that users will hopefully better understand what they are doing - and your 'hopping all over the country' example is a fantastic illustration of why having too many ROUTER nodes in an area is a bad idea. Thanks for your comment on this post; you raise some really good points.

[shalberd]

Hi @erayd

Do you think an addition to the post that expands on the "...will preempt all other roles, and is effectively asserting that it is the best path for all traffic within range." part to further emphasise consideration of what other ROUTER nodes may already be present would be helpful

Yes, that would be a good addition to the text.
Also, we should emphasize that multiple ROUTER nodes that can see each other very well can lead to unnecessarily hops and thus too many ROUTER nodes in any larger area, say 100 x 100 km, should be avoided. That ROUTER_LATE in those mountaintop - near - mountaintop scenarios is much better.

[Ryu945]

@GUVWAF @thebentern @erayd Hello, a small comment

This has the effect of ensuring that ROUTER_LATE will politely 'give way' to other nodes, thereby preserving the normal routing behaviour of the mesh. Other than the obviously higher airtime used, the impact of deploying a ROUTER_LATE node is identical to as if a CLIENT were deployed at that location.

all true, very well-written, thank you.

It is intended to be deployed in sites that do not have the very large coverage footprint typically suited to ROUTER nodes. Sites that are critical for reliable passage of traffic, but known to not be optimally sited for serving the overall mesh, should use ROUTER_LATE."

That is not entirely true ...

The focus of the article should be less about placement of nodes, but rather about rebroadcast priority and early vs normal vs late contention window. That explanation on contention windows and rebroadcoast priority is very well-written in the blog article, thank you.

In our setups, with high mountain nodes widely visible to all sides, think alpine nodes and surrounding great plains, the problem was that ROUTER always rebroadcast and immediately / early grabbed packets from nodes as far away as 80 km. That led to the absurd situation of routers on mountains relaying packets among each other, while not letting clients near to each other and with perfect 0 hop signal quality relay packets to each other.

The green and red nodes are Clients with perfect 0 hop signal quality. The blue nodes were high-altitude, omni-directionally-visible mountain nodes with role Router:

We had to set the mountain nodes to ROUTER_LATE, allowing nodes in a large square-km area to talk to each other directly. Router literally destroyed our Mesh.

So, the explanation that ROUTER_LATE is only applicable to nodes not having a large coverage footprint is definitely not correct in our experience. In fact, we had to set mountain nodes from ROUTER to ROUTER_LATE to avoid a country-wide round-trip tour of Switzerland on perfectly situated nodes with an extreme footprint to all 4 directions over large distances.

See https://github.com/orgs/meshtastic/discussions/24#discussioncomment-14397405

I discussed the issues we faced with @GUVWAF here:

meshtastic/firmware#6904 (comment)

We then followed-up in our Swiss group:

and https://github.com/orgs/meshtastic/discussions/24#discussioncomment-14094052

If your node's coverage footprint is genuinely excellent, please keep it as ROUTER. ROUTER will preempt all other roles, and is effectively asserting that it is the best path for all traffic within range.

I disagree, especially when a node is placed top-notch with long-distance coverage, ROUTER is hurting the mesh. This gets even worse as more than one Router is added.

The delayed rebroadcast window of ROUTER_LATE is essential for mountain nodes with wide, omni-directional, long-distance coverage.

In lateral valleys not accessible from all 4 directions, Client Role is totally sufficient for forwarding inside a valley. That is true even if the node is on either high side of an e.g. V-shaped valley, but not completely on the crest, making east-west or north-south comms possible, but not all 4 directions.

Please note that ROUTER and REPEATER nodes will cause all rebroadcast roles other than ROUTER, REPEATER, ROUTER_LATE and CLIENT_BASE (for packets to / from favourite nodes only) within their coverage area to cancel their own rebroadcasts - so only deploy ROUTER nodes in locations that are actually properly suited to that behaviour.

Because of this, it should be emphasized that especially when long-distance comms from a router node the the surrounding areas is possible, it actually leads to state / province / country-wide round-trips.

We are even of the opinion that the ultra-aggressive (in terms of timing) ROUTER and REPEATER roles are no longer relevant and hurting a mesh more than they help, but that of course is subjective from our experience.

Sorry, not a small comment after all, but very important, I think. In our experience, the agressive timing, combined with the term "router", is suboptimal, in hindsight.

This highlights the directional message sending problem that even Client nodes can have with each other. It is just more obvious and likely with nodes that have higher gain antennas and placed in very good locations. One possibility, but I don't know how you would accomplish this, is to come up with a system where you can determine in which direction the rebroadcasting node is located. If you can determine that the rebroadcast happened in the south direction and the message came from the north direction then you could allow nodes that see themself in a good east and west position to rebroadcast anyway then this would save some bandwidth over the suggested solution of Client nodes ignoring if a Router nodes broadcast.

It wouldn't even have to match the cardinal directions. If you have some way to determine which direction a message is coming in and then assign that as one of the four cardinal directions in this group of four 90 degree blocks, then compare the retransmissions to this made up compass, you can then determine if your in a position where you could add side paths for the message to travel.

I am not sure how this would be done with omnidirectional antennas though.

[shalberd]

it is worth mentioning that device role CLIENT_BASE is only available from the most recent 2.7 alpha version of Meshtastic, not in 2.6.11.

https://github.com/meshtastic/firmware/releases/tag/v2.7.10.94d4bdf

[vpdrums]

I noticed the with new ios update the repeater device role is missing. My Rak module role was a repeater but with this new update show unknown, and there is no repeater mod. The router became the new repeater?

[erayd]

I might have to reconsider my choice here, I don't want to increment that much its utilization.

How much traffic actually exists to / from the nodes that your CLIENT_BASE has set as favourite? For most people, this won't be much - so you may be overestimating how much impact you'll actually have. Those are the only packets that CLIENT_BASE is a mandatory rebroadcaster for.

...is there any way that I can see which path traffic goes?

For flood packets, it goes via every valid path (I.e. every mandatory-rebroadcast node + every CLIENT that isn't suppressed by overhearing another rebroadcast, until the hop limit hits zero). Not just one path. Nodes will only process the packet the first time they receive it; if other copies of it arrive later, they are simply discarded as unwanted duplicates. For unicast packets, they will go via the remembered next hop for the destination node (if known) - and if that doesn't work, they'll fall back to the regular flood logic.

There's no way to view the full path a particular packet took, because that information is not stored in the packet at all. Best you can do is to see the last node that rebroadcast it before you received it. That data is in the packet header as the LSB of the rebroadcaster's own node ID.

There isn't an interface that I'm aware of which will display it for you (including the python CLI tool) - so you'll have to either build something to parse it out, or modify the firmware to print it to the serial console. The latter is likely to be the easiest - just pop a LOG_INFO in there that prints the relay_node field of the meshtastic_MeshPacket struct.

[fzd2vp4nmb-oss]

< How much traffic actually exists to / from the nodes that your CLIENT_BASE has set as favourite? >
Not so much, my internal node (which does not receive any node other than my CLIENT_BASE) does not generate much traffic.
I was more concerned about this: ROUTER-SOUTH broadcasts traffic1, my CLIENT-BASE receives it and re-broadcast it to my CLIENT_MUTE node. By doing so it might collide with some other traffic from other ROUTERs so that ROUTER-SOUTH might not receive them. But this is because I'm still learning quite a bit about the whole process, so I might be mistaken here in considering the wrong way this air-time contention between routers.

< just pop a LOG_INFO in there that prints the relay_node field of the meshtastic_MeshPacket struct. >

Thanks for the hint, I will log data via cli from the serial and see to get this relay_node info (I saw in my case some value like 196); once I have this number, is there a mapping somewhere to see which nodeId relates to that number ? I checked with --info and in the node list I get, I couldn't find that relay node 196 as part of any data element provided by the node list.
Again thanks for your time in clarifying this.

[erayd]

ROUTER-SOUTH broadcasts traffic1, my CLIENT-BASE receives it and re-broadcast it to my CLIENT_MUTE node.

Note that your CLIENT_BASE is only guaranteed to do this for unicast traffic that's addressed to one of your nodes that the CLIENT_BASE has set as favourite. It does not do this for inbound broadcast traffic. For inbound broadcast traffic, it behaves like a regular CLIENT (including cancelling its own rebroadcast if it hears another node rebroadcasting).

...is there a mapping somewhere to see which nodeId relates to that number?

It's literally just the LSB of the rebroadcaster's node ID (i.e. id & 0xFF). To find the node it relates to, compare it to the LSB of other known nodes and see what matches. You can sometimes get collisions (i.e. more than one known node matches) - if that happens, you'll need to use a bit of common sense to determine which of the matching nodes it's likely to be (e.g. if one of them is on the other side of a mountain from your, it's probably not that one).

[fzd2vp4nmb-oss]

< Note that your CLIENT_BASE is only guaranteed to do this for unicast traffic that's addressed to one of your nodes that the CLIENT_BASE has set as favourite. It does not do this for inbound broadcast traffic. >

Ok, now this clarifies more. So it's basically handling direct messages. So when a node sends its telemetry or other info like nodeinfo etc, that I suppose are broadcasted packets, it then treat them as a CLIENT role.
Perfect, it makes sense. So it's not really that much that I will interfere; at the moment I didnàt use much of direct message, therefore it never acted as ROUTER.

< It's literally just the LSB of the rebroadcaster's node ID (i.e. id & 0xFF). >
Ok thanks for this, I will play little bit with it.
Thanks again, I needed all these clarifications.
Regards, Armando

[erayd]

Ok, now this clarifies more. So it's basically handling direct messages.

Not just direct messages. It's guaranteed to rebroadcast any packet that is addressed either to or from one of its favourited nodes. That will include things like outbound telemetry from those favourite nodes. However, direct messages is certainly the primary use-case for it.

When other nodes elsewhere on the mesh send their own telemetry packets, they are sent to 0xFFFFFFFF (the broadcast address), from their own node ID. Your CLIENT_BASE will behave like a regular CLIENT for those, because it will have neither the distant sender nor the broadcast address set as a favourite.

Ok thanks for this, I will play little bit with it.
Thanks again, I needed all these clarifications.

Good luck, and you are most welcome 🙂

If you're not already a member of the Meshtastic Discord, I'd recommend that you join - there are a ton of helpful folks there who will assist you with troubleshooting / understanding things as needed.