ROSbot XL MicroROS/RTOS error after some time

Hi,

Our ROSbot XL functions normally when the robot docker containers are launched.

However, after some time, the robot eventually runs into the issue that the topic /cmd_vel is not present in the robot. This issue was tracked down to the microROS issue because the LED 2 flashes red.

The only solution at this point is manually pressing the power button and turn it back on.

I want to prevent this scenario or solve it through the onboard PC (rebooting it does not fix the issue, nor restarting the docker containers).

Do you have any suggestions on how to do this?

Thank you.

As a followup, this is the output I get for the rosbot-xl container. It seems like the controller_manager has issues finding the data coming from the micro controller?

[INFO] [launch]: All log files can be found below /root/.ros/log/2024-02-05-08-57-44-199737-rosbot-xl-1
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [ros2_control_node-1]: process started with pid [68]
[INFO] [robot_state_publisher-2]: process started with pid [70]
[INFO] [spawner-3]: process started with pid [72]
[INFO] [ekf_node-4]: process started with pid [74]
[INFO] [scan_to_scan_filter_chain-5]: process started with pid [78]
[robot_state_publisher-2] [INFO] [1707123464.717346239] [robot_state_publisher]: got segment antenna_connector_link
[robot_state_publisher-2] [INFO] [1707123464.718371045] [robot_state_publisher]: got segment antenna_link
[robot_state_publisher-2] [INFO] [1707123464.719058987] [robot_state_publisher]: got segment base_link
[robot_state_publisher-2] [INFO] [1707123464.719688069] [robot_state_publisher]: got segment body_link
[robot_state_publisher-2] [INFO] [1707123464.720785777] [robot_state_publisher]: got segment cover_link
[ros2_control_node-1] [INFO] [1707123464.720802930] [resource_manager]: Loading hardware 'imu' 
[robot_state_publisher-2] [INFO] [1707123464.721910590] [robot_state_publisher]: got segment fl_wheel_link
[robot_state_publisher-2] [INFO] [1707123464.722496801] [robot_state_publisher]: got segment fr_wheel_link
[robot_state_publisher-2] [INFO] [1707123464.723171238] [robot_state_publisher]: got segment imu_link
[robot_state_publisher-2] [INFO] [1707123464.723354913] [robot_state_publisher]: got segment laser
[robot_state_publisher-2] [INFO] [1707123464.723369246] [robot_state_publisher]: got segment rl_wheel_link
[robot_state_publisher-2] [INFO] [1707123464.723380779] [robot_state_publisher]: got segment rr_wheel_link
[robot_state_publisher-2] [INFO] [1707123464.723391079] [robot_state_publisher]: got segment slamtec_rplidar_s1_link
[ros2_control_node-1] [INFO] [1707123464.729439545] [resource_manager]: Initialize hardware 'imu' 
[ros2_control_node-1] [INFO] [1707123464.729488345] [RosbotImuSensor]: Initializing
[ros2_control_node-1] [INFO] [1707123464.739000802] [resource_manager]: Successful initialization of hardware 'imu'
[ros2_control_node-1] [INFO] [1707123464.739154669] [resource_manager]: Loading hardware 'wheels' 
[ros2_control_node-1] [INFO] [1707123464.739421359] [resource_manager]: Initialize hardware 'wheels' 
[ros2_control_node-1] [INFO] [1707123464.739438293] [RosbotSystem]: Initializing
[ros2_control_node-1] [INFO] [1707123464.739496911] [RosbotSystem]: Joint 'fl_wheel_joint' found
[ros2_control_node-1] [INFO] [1707123464.739585513] [RosbotSystem]: Joint 'fr_wheel_joint' found
[ros2_control_node-1] [INFO] [1707123464.739609302] [RosbotSystem]: Joint 'rl_wheel_joint' found
[ros2_control_node-1] [INFO] [1707123464.739618162] [RosbotSystem]: Joint 'rr_wheel_joint' found
[ros2_control_node-1] [INFO] [1707123464.748794784] [resource_manager]: Successful initialization of hardware 'wheels'
[ros2_control_node-1] [INFO] [1707123464.749566922] [resource_manager]: 'configure' hardware 'imu' 
[ros2_control_node-1] [INFO] [1707123464.749586067] [RosbotImuSensor]: Configuring
[ros2_control_node-1] [INFO] [1707123464.749594558] [resource_manager]: Successful 'configure' of hardware 'imu'
[ros2_control_node-1] [INFO] [1707123464.749602823] [resource_manager]: 'activate' hardware 'imu' 
[ros2_control_node-1] [INFO] [1707123464.749608492] [RosbotImuSensor]: Activating
[ros2_control_node-1] [WARN] [1707123464.752902044] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] 1707123464.964250 [0]    spawner: config: //CycloneDDS/Domain/General: 'NetworkInterfaceAddress': deprecated element (file:///var/tmp/husarnet-cyclone.xml line 6)
[ros2_control_node-1] [WARN] [1707123465.253265249] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123465.753480730] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123466.253847809] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123466.754244648] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123467.041512102] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123467.254449343] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123467.754721436] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123468.254958186] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123468.755262315] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123469.068669819] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123469.255557180] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123469.755823571] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123470.256095982] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123470.756424133] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123471.096127834] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123471.256605214] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123471.756859322] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123472.257082454] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123472.757389910] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123473.122654365] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123473.257666221] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123473.757864832] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123474.258037608] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123474.758249994] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123475.143462268] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123475.258465848] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123475.758739761] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123476.258988622] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123476.759197098] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123477.171335542] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123477.259412008] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123477.759689876] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123478.259989411] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123478.760269749] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123479.199450620] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123479.260581181] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123479.761102951] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123480.261371873] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123480.761598661] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123481.227619051] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123481.261863087] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123481.762123770] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123482.262382074] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123482.762583677] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123483.255765481] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123483.262790484] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123483.763005957] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123484.263230881] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123484.763398213] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123485.263613595] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123485.279171891] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123485.763868276] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123486.264077786] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123486.764330101] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123487.264599481] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123487.307334221] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123487.764869317] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123488.265112749] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123488.765384452] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123489.265650492] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123489.335402558] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123489.765930010] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123490.266145641] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123490.766445312] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123491.266656202] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123491.363027993] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123491.766924711] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123492.267138012] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123492.767416225] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123493.267687626] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123493.390311273] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123493.767952308] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123494.268214658] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123494.768561691] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123495.268762071] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123495.423106195] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123495.768950527] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123496.269168874] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123496.769437349] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123497.269708119] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123497.449739348] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
[ros2_control_node-1] [WARN] [1707123497.770021009] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123498.270328757] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123498.770641787] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[ros2_control_node-1] [WARN] [1707123499.270931459] [RosbotImuSensor]: Feedback message from imu wasn't received yet
[spawner-3] [INFO] [1707123499.476702773] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available

Hi @rodrigo55,

However, after some time, the robot eventually runs into the issue that the topic /cmd_vel is not present in the robot. This issue was tracked down to the microROS issue because the LED 2 flashes red.

What do you mean saying “some time”? Does it happen when the containers are restarted?
Does the first run of the containers on new rebooted robot work?

Best regards,
Jakub Delicat

Thank you for your answer,

What I mean by “after some time” can actually be days later.

My intention with this robot is to have it operational 24/7. I’ve attached it to continuous power for this.

So I will turn the robot on and leave it do its thing. Eventually, the issue with the disappearing /cmd_vel and the red LED appears.

Once that happens, restarting the containers does not solve it, nor does rebooting the PC (so the first run of the rebooted robot doesn’t work either). The only way to get the robot operational again is to manually press the power button to restart it.

Once I manually restart it. Everything runs smoothly again.

Thank you for describing the issue. It seems like the issue is with the ROSbot XL firmware or microros agent. We will investigate that.

The quick workaround to restart the firmware without the power restart:
You can reset the microcontroller with MCU reset button at ROSbot’s XL rear panel during the containers run it reinitializes the microros entites and the green LED should appear.