How should we handle multiple publishers simultaneously posting requests?

[mxgrey]

If multiple unrelated publishers are posting different goals to ~/destination/goal at the same time, then there is ambiguity about what goal the robot is actually meant to pursue.

It is important to consider that different subscribers may receive messages in a different order if the messages are coming from different publishers. Therefore if we naively accept the "latest" (most recently received) request as the "true" request then different subscribers could fall out of sync with each other.

If we put a timestamp inside the message definitions and judge the "latest" message on the most recent timestamp then there are a few issues:

• Different system clocks may fall out of sync (I've seen hours and up to a day of sync issues), creating substantial bias in which message is most "recent"
• Publishers could spoof their timestamp or forget to fill it in
• Timestamps could result in a draw, although the probability of this is extremely low when clocks have nanosecond precision

Note that some RMW implementations provide source timestamps in the message metadata, but this is not guaranteed to be provided and shares some of the same potential issues as an internal timestamp, especially the first problem of system times falling out of sync.

I would recommend that we make it a system-level requirement that each ~/destination/goal topic should only have one publisher publishing to it.

• Different instances of the topic may have different publishers, e.g. security_bot_1/destination/goal could have a different publisher than cleaning_bot_3/destination/goal
• The same publisher can publish to multiple different topic instances, e.g. security_bot_schedule can publish to security_bot_1/destination/goal, security_bot_2/destination/goal, etc...

However I am open to other ideas on how to settle this, for example queuing up incoming requests or putting a string requester_id in the message definition and having a priority scheme to choose between the publishers.

[sauk2]

@mxgrey What do you think about having a robot-level resource lock. This would involve applying a lock directly to the robot, ensuring that only the authorized publisher can issue commands to its /destination/goal topic.

Here's how it could work:

• A lock manager would expose /acquire_robot_lock and /release_robot_lock services for each robot configured to consume commands from multiple publishers.
• Publishers would first acquire a lock through this service, receiving a lock_handle_id.
• The plan server would then only accept requests for that robot from a publisher holding an active lock_handle_id.
• These locks would have timeouts and require periodic renewal to prevent stale locks if the node crashes.

This lock service would be optional,

• If the acquire_robot_lock service is available and enabled for a robot, publishers must use it to command that robot.
• If the service is not available, the system would default to a single designated publisher as recommended by you.

Let me know your thoughts!

[mxgrey]

@sauk2 I think that's a good idea, especially if we can generalize it beyond robots.

A few things to consider:

• Should we support preempting? E.g. An operator wants to take over and teleoperate the robot, so the task management system should lose the right to command the robot until the operator is finished.
• How about prioritization? E.g. Emergency alarm takes priority over task management, and operators take priority over emergency alarm.
• Instead of lock_handle_id, what if we have the lock server indicate which session_id (UUID) is currently approved for execution?

[sauk2]

@mxgrey Thanks for the feedback! I agree that it would be great to generalize this beyond robots to other shared resources.

• Should we support preempting? E.g. An operator wants to take over and teleoperate the robot, so the task management system should lose the right to command the robot until the operator is finished.
• How about prioritization? E.g. Emergency alarm takes priority over task management, and operators take priority over emergency alarm.

My initial thought was that the preemption and prioritization logic could lie inside the resource lock manager.

• For prioritization, requestors could include a priority level in their acquire_lock request. The lock manager would arbitrate and grant locks based on priorities.
• For preemption, the lock manager could be designed to revoke a current lock and grant it to the new requestor. If a higher-priority request arrives, the previously authorized system could be notified (by its renew lock failing or through a dedicated notification from the lock manager) that it no longer holds a valid lock.

I am thinking this can also prove to be a solution to #1. Let me know what you think!

• Instead of lock_handle_id, what if we have the lock server indicate which session_id (UUID) is currently approved for execution?

I think this is a great idea! This would mean the lock manager would be effectively granting permission for a specific session to send commands.