Losing /scan data and unhealthy containers

[daria]

Hello,

long story short, after about a dozen minutes the /scan topic stops being published. Then sometime retsart a dozen minutes later but I lose /scan_filtered definitely

I have a Rosbot XL, on ubuntu 22.04 (humble) I use a Slamtec S2 and rosbot_xl with the following config and images:

  rosbot-xl:
    image: husarion/rosbot-xl:humble-0.11.5-20240614 #0.8.2-20230913
    container_name: rosbot-xl
    <<: *common-config
    command: ros2 launch rosbot_xl_bringup bringup.launch.py lidar_model:='slamtec_rplidar_s2' mecanum:=${MECANUM:-True} 
rplidar:
   image: husarion/rplidar:humble
   <<: *common-config
   privileged: True
   devices:
   - /dev/ttyUSB1:/dev/ttyRPLIDAR
   command: ros2 launch sllidar_ros2 sllidar_s2_launch.py serial_baudrate:=${LIDAR_BAUDRATE:-1000000} inverted:='false' serial_port:='/dev/ttyRPLIDAR'

Both rosbot-xl and husarion-rplidar-1 are considered “unhealthy” by docker, here are the logs:

Docker logs rosbot_xl

[ekf_node-5] Failed to meet update rate! Took 16.410924095999998684seconds. Try decreasing the rate, limiting sensor output frequency, or limiting the number of sensors.
[ekf_node-5] Failed to meet update rate! Took 24.260967296000000459seconds. Try decreasing the rate, limiting sensor output frequency, or limiting the number of sensors.
[ekf_node-5] Failed to meet update rate! Took 16.295909760000000688seconds. Try decreasing the rate, limiting sensor output frequency, or limiting the number of sensors.
[ekf_node-5] Failed to meet update rate! Took 13.185206816000000885seconds. Try decreasing the rate, limiting sensor output frequency, or limiting the number of sensors.
[ekf_node-5] Failed to meet update rate! Took 6.8889963200000003951seconds. Try decreasing the rate, limiting sensor output frequency, or limiting the number of sensors.
[ekf_node-5] Failed to meet update rate! Took 4.510549024000000351seconds. Try decreasing the rate, limiting sensor output frequency, or limiting the number of sensors.
[ekf_node-5] Failed to meet update rate! Took 5.5649965440000004335seconds. Try decreasing the rate, limiting sensor output frequency, or limiting the number of sensors.

for husarion_rplidar_1

[sllidar_node-1] [INFO] [1747038801.409811384] [sllidar_node]: SLLidar running on ROS2 package SLLidar.ROS2 SDK Version:1.0.1, SLLIDAR SDK Version:2.1.0
[sllidar_node-1] [INFO] [1747038801.430050616] [sllidar_node]: SLLidar S/N: AA32E18BC7E49CCDA3EB9AF43EEF4C73
[sllidar_node-1] [INFO] [1747038801.430168312] [sllidar_node]: Firmware Ver: 1.02
[sllidar_node-1] [INFO] [1747038801.430200216] [sllidar_node]: Hardware Rev: 18
[sllidar_node-1] [INFO] [1747038801.432086616] [sllidar_node]: SLLidar health status : 0
[sllidar_node-1] [INFO] [1747038801.432175000] [sllidar_node]: SLLidar health status : OK.
[sllidar_node-1] [INFO] [1747038801.600701848] [sllidar_node]: current scan mode: DenseBoost, sample rate: 32 Khz, max_distance: 18.0 m, scan frequency:10.0 Hz,

I still receive /odometry_filtered but /scan vanishes from time to time over a couple to a dozen minutes, no log helps me figure out why. the USB is still detected and still the same name.
It is not usable as it is for navigation right now.

My supposition so far is that the lidar request too much current to the jetson Orin Nano? But that is something you would not have overlooked while conceiving the robot.

I checked with top, the CPU is used at 20%, however there are boot of heavy lagging when sending commands to the robot. The jetson is using 15W

The robot serial number is: 4ade2c

[daria]

Could the issue be due to a CPU overload?
running jtop:

All the hearts are usually at about 10% but they have spikes going up to 99% (and I do not know why 2 hearts are off! I suspect it is to stay at a 7W usage)

The only things that should be running are the dockers (microros, rplidar and rosbot_xl)
and top tells us the details:

 14740 husarion  20   0 1916560  50036  21156 S  12.5   0.6   8:45.24 ros2_control_no                                                                                                            
  14722 husarion  20   0  779916  29000  14456 S   6.2   0.4   6:24.26 sllidar_node

(please if you ask me to run some tests, specify the commands)

[daria]

Oh yes. Stopping and restrating the dockers work for a few more dozen minutes… but that is not a solution.

[RafalGorecki]

Hello @daria,
I have two suggestions to check.

1. From what I remember, you can set power usage settings on Jestosn, since 2 cores are disabled. This may suggest that Jetson is running in power saving mode. Please verify if this is the case.
2. Check if running only rplidar (without rosbot) improves the frequency of sending scan messages (ros2 topic hz /scan)
3. Try doing docker compose pull. If a new rplidar image has been downloaded, this should solve the unhealthy issue, but I would not expect any improvement in efficiency.
4. You mentioned microros, it is not needed for versions above 0.9.0, because it is included in rosbot_ros
5. Can you send a picture, how did you connect rplidar

Regards

[daria]

Yes, you can see the power mode in the image:

sudo nvpmodel -q
NV Power Mode: 7W

So 7W is the power mode. Running the 15W or higher crashed the jetson if we were using the robot battery with all the USB devices plugged in (not considering sensors). It took us a while to understand what the issue was while installing the Jetson image (and consistently failing).
Should we try the 15W again?

2. I did not notice issues with the /scan average rate. Tried again with and without rosbot_xl and it is consistently around 10Hz (which is what is requested of it)

3. Indeed the rplidar was not the latest version available

4. good to know, I’m removing it (don’t see much difference though)

5. You want a picture of the USB connection?
   It’s connected through the UART adaptator → Bus 001 Device 006: ID 10c4:ea60 Silicon Labs CP210x UART Bridge

Would you know a terminal command to check if we actually receive the data from the lidar? Because /scan stops emitting, but is it due to the lidar not sending data anymore? I’m not sure. There are no ros log errors.

Oh yes, note that the hz /scan has been verified in the robot, remotely it is a lower frequency, but that is to be expected.

[daria]

here is the whole connection setup (removed the camera USB connection, the lidar was below)

Here is the bridge connection:

Note that we were not provided a 5V cable (but I assumed that was normal, as were would we plug it?). The connector above seems well linked (red cable on 5V)

[daria]

I switched to a 15W power operation and the lidar hasn’t been lost for over 20minutes.
So the issue might have been the lidar requiring more current than what the 7W powered jetson could deliver.
Some more testing and this issue can be closed. Thanks for the hint!
(if that is correct, the problem was hardware, not software)

[daria]

The power input was the issue. This thread can be closed

[RafalGorecki]

I’m glad the issue has been resolved. Out of curiosity, which power supply are you talking about, and what would you @daria need to see in the documentation to prevent this issue from occurring?

[daria]

sorry for the delay, I missed the notification.
Thanks for asking.

I set the jetson in power optimisation mode (7W), which was probably not enough to power both the jetson and the lidar. Passing it to 15W solved the issue.

I would probably mention it in : SLAMTEC - RPlidar Series | Husarion
The two relevant places I found information about Lidar was on that link and : SLAM | Husarion

I would probably risk repeating myself and have a dedicated tutorial on how to set up each piece of equipment (with the command line to add it to the compose.yaml file, etc., so it appears in the TF tree), and ideally, how to modify the code to adjust the equipment placement on the robot.

Have a great day

[RafalGorecki]

Thank you, so much, for your feedback!