several issues using the correct executor

[marc-hanheide]

@francescodelduchetto and I are a bit lost what the "correct" executor/scheduler would be to use for our museum case.

The problem: We have so far tried to use https://github.com/strands-project/strands_executive/blob/kinetic-devel/task_executor/launch/task-scheduler-top.launch as a simple scheduler. However, that one seems to be deprecated, and lacking recent features (e.g. time critical tasks). Also with that one, we had the following issue:

@francescodelduchetto submitted in private issue https://github.com/LCAS/LindIMP/issues/40:

Basically, a task was running and I was trying to cancel it but I wasn't able to do so...bcs of this error I presume @marc-hanheide

[INFO] [1542370813.904951]: Concurrent state 'MONITORING' returned outcome 'preempted' on termination.
[WARN] [1542370813.923710]: State 'MONITORING' in concurrence did not service preempt.
[INFO] [1542370813.924395]: Concurrent Outcomes: {'MONITORED': 'succeeded', 'MONITORING': 'preempted'}
[INFO] [1542370813.924950]: Action terminated with outcome succeeded
[INFO] [1542370813.931177]: State machine transitioning 'TASK_EXECUTION':'succeeded'-->'TASK_SUCCEEDED'
[INFO] [1542370813.932638]: Preempt requested on state machine before executing the next state.
[INFO] [1542370813.933305]: Last state 'TASK_EXECUTION' did not service preempt. Preempting next state 'TASK_SUCCEEDED' before executing...
[ERROR] [1542370813.941010]: InvalidUserCodeError: Could not execute state 'TASK_SUCCEEDED' of type '<task_executor.sm_base_executor.TaskSucceeded object at 0x7f1a2b95bc$
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/smach/state_machine.py", line 247, in _update_once
    self._remappings[self._current_label]))
  File "/localhome/lcas/workspaces/museum_ws/src/task_executor/src/task_executor/sm_base_executor.py", line 79, in execute
    self.executor.task_succeeded(completed)
  File "/localhome/lcas/workspaces/museum_ws/src/task_executor/scripts/scheduled_task_executor.py", line 139, in task_succeeded
    self.task_complete(task)
  File "/localhome/lcas/workspaces/museum_ws/src/task_executor/scripts/scheduled_task_executor.py", line 135, in task_complete
    self.execution_schedule.task_complete(task)
  File "/localhome/lcas/workspaces/museum_ws/src/task_executor/src/task_executor/execution_schedule.py", line 118, in task_complete
    assert self.current_task is not None
AssertionError

Exception in thread Thread-89:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/smach/state_machine.py", line 359, in execute
    container_outcome = self._update_once()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/smach/state_machine.py", line 254, in _update_once
    + traceback.format_exc())
InvalidUserCodeError: Could not execute state 'TASK_SUCCEEDED' of type '<task_executor.sm_base_executor.TaskSucceeded object at 0x7f1a2b95bc10>': Traceback (most recent $
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/smach/state_machine.py", line 247, in _update_once
    self._remappings[self._current_label]))
  File "/localhome/lcas/workspaces/museum_ws/src/task_executor/src/task_executor/sm_base_executor.py", line 79, in execute
    self.executor.task_succeeded(completed)
  File "/localhome/lcas/workspaces/museum_ws/src/task_executor/scripts/scheduled_task_executor.py", line 139, in task_succeeded
    self.task_complete(task)
  File "/localhome/lcas/workspaces/museum_ws/src/task_executor/scripts/scheduled_task_executor.py", line 135, in task_complete
    self.execution_schedule.task_complete(task)
  File "/localhome/lcas/workspaces/museum_ws/src/task_executor/src/task_executor/execution_schedule.py", line 118, in task_complete
    assert self.current_task is not None
AssertionError

So, maybe we should be using the MDP scheduler? That one, however, in the current version uses the a door config file:

strands_executive/task_executor/launch/mdp-executor.launch

Lines 9 to 11 in 74d4d03

	<include file="$(find mdp_plan_exec)/launch/mdp_plan_exec_extended.launch">
	<arg name="door_config_file" value="$(arg door_config_file)"/>
	</include>

And it doesn't seem to work properly. Can @bfalacerda or @hawesie suggest what the correct launch file would be for a "simple" (no doors, no adaptation, no bells and whistles), but robust and full-featured scheduling? I feel we are doing something stupid here. If it helps, I can invite you guys to the private repo to understand our situation better. Simple first questions: Which launch file should we be using?

[hawesie]

Hmm... we run this basic launch file all the time with no problems. If you want no adaptation then take out fremen from top nap and add the following instead:

 <node pkg="topological_navigation" name="speed_based_prediction" type="speed_based_prediction.py" output="screen" respawn="true"/>

[hawesie]

So, assuming you're running this launch file, what's going wrong?

[marc-hanheide]

for starters, we are not sure which task_executor to actually launch ;-) So, if you could tell us which one to use, then we say what's not working with it. @francescodelduchetto can say more about it, I'm sure, but if I recall, launching the MDP one the first hurdle was that it requested some door config...

[hawesie]

Run mdp-executor.launch which is the MDP one. The door config defaults to empty which works just fine normally.

[francescodelduchetto]

Yes, the mdp-executor.launch seems to work fine also for us.

I just tested it and got this error/warning fo the doors

WARN] [1542384778.726317]: Error loading yaml config for door waits: [Errno 2] No such file or directory: ''. Using default values for all doors.

and yet it seems to schedule and start demanded tasks properly.

[hawesie]

Yep, it's a bit cryptic, but Using default values for all doors looks good.

[marc-hanheide]

Hooray, sorry for bothering you @hawesie I thought we tried that one and it wouldn't run anything... So, that's the recommended one. Perfect. Thanks.

[marc-hanheide]

We are still struggling with it, sorry. Whenever we submit a task, it ends with "Adding Doors" and then hangs, and also other services we call never return. There is an explicit_doors flag that is true in this setup, but I don't quite understand all the stuff it does. I would feel more comfortable if we could turn it off. But @hawesie said it shouldn't have any impact as long as the file isn't found, but it still seems to call a /door_predict service which we don't have, I think. Any idea how we could debug best. The old launch files (the non-extended mdp_executor, for instance) don't start at all. I feel we are still launching it all wrong.

[francescodelduchetto]

The mdp executor actually remains hung every time here:

strands_executive/task_executor/scripts/mdp_task_executor.py

Line 557 in d3b5699

service_response = estimates_service(request)

that service call never returns. It's odd how it sometimes works (like 3 days ago) and others not. Any idea on why this is happening?

[hawesie]

Could it be that Fremen is crashing and not providing predictions? Sorry away from computer so can't look in detail right now