AMCL never publishes map->odom transform (persistent "queue is full" for laser scans) — husarion_ugv_autonomy_ros, real Panther hardware

1

AMCL never publishes map->odom transform (persistent “queue is full” for laser scans) — husarion_ugv_autonomy_ros, real Panther hardware

Environment

  • Robot: Husarion Panther, real hardware

  • Repo: husarion/husarion_ugv_autonomy_ros (ros2 branch, cloned fresh)

  • ROS 2 Jazzy

  • User Computer: running as a VMware Workstation Pro VM (Ubuntu 22.04.5), bridged network directly to the Panther’s switch (confirmed 0% packet loss, <1ms ping to 10.15.20.2)

  • LiDAR: RPLiDAR S3, connected via USB passthrough to the VM, with a custom scan_filter.py node that blanks out the rear 180° of the scan (robot chassis is in the way) and republishes on /scan_filtered

  • SLAM=False, using a pre-built, verified map (map_server loads it correctly: “Read map /maps/map.pgm: 963 X 587 map @ 0.05 m/cell”)

Symptom
AMCL activates normally (lifecycle: Configuring → Activating → “Setting pose (…): 0.000 0.000 0.000” → “Managed nodes are active”), but it never publishes the panther/map → panther/odom transform. The following repeats indefinitely, every ~2.5s, and never self-resolves:

[panther.amcl]: Message Filter dropping message: frame 'laser' at time X for reason 'discarding message because the queue is full'
[panther.global_costmap.global_costmap]: Timed out waiting for transform from panther/base_link to panther/map to become available, tf error: Invalid frame ID "panther/map" ... frame does not exist
[panther.global_costmap.global_costmap]: Failed to activate global_costmap because transform ... did not become available before timeout
[panther.lifecycle_manager_navigation]: Failed to bring up all requested nodes. Aborting bringup.

/panther/particlecloud is never published, confirming AMCL never successfully processes a single scan.

What I’ve confirmed working (individually, via direct tf2_echo / ros2 topic hz tests)

  • panther/base_link → laser (our static_transform_publisher): stable, correct values every time

  • panther/odom → panther/base_link (from ekf_filter): stable, correct values every time, ~50Hz

  • /scan_filtered: publishing at ~20-40Hz, correct frame_id (“laser”), sane range data (also tested with raw unfiltered /scan — same result)

  • global_frame_id/odom_frame_id/base_frame_id on amcl: correctly set to panther/map/panther/odom/panther/base_link

  • No duplicate nodes on the graph (checked ros2 node list for dupes)

  • use_sim_time on amcl: false (correct for real hardware)

  • Time is synced between the VM and the Panther’s onboard computer (checked actual message header.stamp values from both /scan_filtered and /panther/odometry/filtered — sub-300ms difference, not growing over time)

  • Only one publisher/subscriber pair on /scan_filtered, correctly matched QoS (we found and fixed an initial RELIABLE vs BEST_EFFORT mismatch between our filter node and amcl/costmap subscriptions — this did NOT fix the issue, but is a legitimate config error we corrected)

What I’ve tried, with no effect on the core issue

  • Increasing transform_tolerance to 2.0 for amcl, local_costmap, and global_costmap (baked into nav2_params.yaml before first activation, not just live param set)

  • Publishing an explicit /panther/initialpose (both zero and non-zero covariance) via ros2 topic pub

  • Full stack restarts (LiDAR, filter, static TF, and Nav2 all together, clean order)

  • Rebuilding the Docker image (docker compose build --no-cache) from a completely fresh git clone with minimal .env changes

  • Restarting the Panther’s onboard ROS driver container

  • Switching the VM’s virtual NIC from e1000 to vmxnet3 (better network stack, confirmed lower-level, direct route to the robot)

  • Fixing the QoS mismatch mentioned above

Question
Is there a known interaction in the husarion_ugv_autonomy_ros nav2_params.yaml template between AMCL’s laser message_filter and a custom/remapped scan topic (as opposed to the default sensor pipeline) that could cause this specific permanent “queue is full” deadlock? Or is there a recommended way to get debug-level logs specifically from the amcl node (RCUTILS_LOGGING_SEVERITY_THRESHOLD=DEBUG in docker/.env did not seem to produce additional amcl-specific output) so I can see what’s happening internally when it tries (and fails) to process a scan?

Happy to provide full logs, ros2 topic info --verbose dumps, or anything else needed to help diagnose this.

2

Hi @emilijas

I’m missing information that could be valuable for resolving these issues.

  1. AMCL would like to receive a map. Do you have a map created with Slam Toolbox or another compatible tool, and is the map topic available?

  2. I assume you’re running AMCL on a User Computer. Please check if time synchronization has been set, and the time is correct on both machines. User Computer Setup Guide | Husarion

Let me know if you need assistance with any of the above issues or if you discover any new information.

3

Hi,

1. Map: Yes, we have a map. It was created earlier using slam_toolbox (via this repo’s just start-hardware navigation with SLAM=True), then saved with nav2_map_server’s map_saver_cli. We’re currently running with SLAM=False and the pre-built map. Logs confirm it loads correctly:

[map_io]: Read map /maps/map.pgm: 963 X 587 map @ 0.05 m/cell
[panther.amcl]: Subscribed to map topic.
[panther.amcl]: Received a 963 X 587 map @ 0.050 m/pix

So the map itself and the map topic are confirmed working.

2. Time synchronization: Yes, this is running on a User Computer (in our case, a VM on a separate machine, bridged directly to the Panther’s network). Here is exactly how we’ve been handling time sync so far:

At the start of each session, before launching anything, we manually pull the Built-in Computer’s current time and set it as a one-time snapshot on the User Computer:

bash

sudo date -s "$(ssh husarion@10.15.20.2 'date')"

We verified this brings both clocks within 1-3 seconds of each other (checked via date +%s on both sides). We also went a step further and compared the actual header.stamp values of live messages from both sides (/scan_filtered from the User Computer vs /panther/odometry/filtered from the Built-in Computer) — the difference stayed stable around 100-300ms over several minutes, well within Nav2’s transform_tolerance (which we increased to 2.0s).

However, we were not running a continuous NTP-style sync (no Chrony) — just this one-time date -s snapshot per session. We just found your official User Computer Setup Guide recommending Chrony as a continuous client against the Built-in Computer’s NTP server, and we’re setting that up now, since we hadn’t tried that specific approach.

Given all of the above (map confirmed loading, individual TF links all verified working via tf2_echo, matched QoS between our scan filter and amcl/costmap subscriptions, transform_tolerance increased), do you know of any other place in the AMCL/Nav2 pipeline where sub-second-level time precision would matter in a way our one-time sync could still be insufficient for, even though the drift we measured directly was small and stable (not growing)?

Additional context that might help diagnose this: at one point during our testing, the system was actually working correctly — I could see the robot correctly positioned on the map, successfully set its initial pose via “2D Pose Estimate”, and sent a goal pose. That first navigation attempt didn’t execute, but only because the E-stop was active (Nav2 correctly reported “E-stop activated. Halting navigation.” and aborted the goal) — that was expected/correct behavior on our part, not a bug.

After that, while everything was still running live, I ran vmware-toolbox-cmd timesync enable to let VMware Tools continuously sync the VM’s clock with the Windows host. This caused an immediate, visible CycloneDDS error across all running nodes:

sequence size exceeds remaining buffer

From that point on, the panther/map -> panther/odom transform has never recovered, even after resetting E-stop, restarting every node individually, restarting the whole stack, and eventually rebooting the VM entirely. So whatever broke seems to have been triggered by that abrupt clock jump hitting an already-live DDS session, though we haven’t been able to identify exactly what got corrupted or why a full VM/container restart doesn’t reset it. We tired to go back with the steps I send but nothing is helping

4

Thank you @emilijas for checking this. It’s very valuable information that it worked with SLAM=True.
I’m not familiar with VMware toolbox, so it might be worth temporarily disabling it: vmware-toolbox-cmd timesync disable. Can you verify this?

I’ll run your case in a simulation and make sure that AMCL works in the simulation.

5

Hi again,

Wanted to close the loop on this thread. We finally resolved it by moving the User Computer off the VM entirely, onto a physical Jetson (JetPack 6, Ubuntu 22.04, ROS 2 Jazzy). Even on bare metal, the exact same “queue is full” symptom reproduced at first, which told us it genuinely wasn’t a virtualization-only issue.

The real fix turned out to still be time precision, but our earlier one-time date -s snapshot approach was insufficient. We set up Chrony properly on both sides, continuous NTP client/server, Jetson as a local server (with a local stratum fallback so it stays a valid source even without internet), Panther’s built-in computer as its client, and critically added makestep 1.0 -1 on both, so any offset over 1 second is stepped immediately instead of only being slowly slewed. Once chronyc tracking on both sides actually showed Leap status: Normal (not just a coincidentally close date reading), AMCL came up cleanly, map -> odom published continuously, no more dropped messages.

So it really was time sync after all, just needing a properly running, immediately-stepping NTP pair rather than a manual snapshot, even outside a VM.

Thanks again for pointing us toward the VMware Tools timesync as a lead, that pushed us to look harder at time sync overall, even though the final fix ended up being broader (and needed on native hardware too).

For anyone hitting this later: worth checking chronyc tracking for a real “Normal” leap status on both sides, not just that date looks close, since a dead or never-synced chronyd can still coincidentally show a matching timestamp.

Also I have one more question I will make one more post for this but maybe you can help with that also:

Separate question, this time about navigating narrow passages. Our robot’s footprint requires inflation_radius to be at least ~0.45 (matching the inscribed radius Nav2 reported), otherwise bringup fails with the “inflation radius smaller than inscribed radius” error. That’s fine for most of our space, but we have a few passages only about 1.1-1.2m wide, and we’d like the robot to reliably drive straight through them with something like 50-60mm clearance on each side, similar to how industrial AMRs handle fixed narrow corridors.

A few things we’re trying to understand before we build anything:

  1. Is inflation_radius truly bounded below by the inscribed radius for bringup to succeed, or is there a supported way around that specifically for tight passages?

  2. Is a Costmap Filter (Keepout Filter / mask) the recommended approach for defining a narrower “allowed corridor” through a specific passage while keeping normal inflation everywhere else, or is there a more standard pattern in the husarion-ugv-autonomy stack for this?

6

I’m glad you managed to find the source of the problem, and thank you for sharing your observations with the community.

I’ll add one suggestion: you wrote that your User Computer is configured with JetPack 6, Ubuntu 22.04, and ROS 2 Jazzy. JetPack 7.2, which is based on Ubuntu 24.04, has recently been released; working with ROS 2 Jazzy should be easier then. One note: if it’s a ZED BOX, I don’t know if the ZED camera drivers are compatible with the newer OS version.

I’ll answer your question about driving in narrow spaces in a new thread.