Project Overview
The core objective of this project is to extend the capabilities of ROS2 Nav2 by integrating Large Language Models (LLMs) to enable Semantic Navigation. By allowing the robot to perceive and understand objects in its surroundings, it can move beyond coordinate-based goals to context-aware task execution.
Vision Strategy & Spatial Database: I implemented a hybrid approach to perception. I integrated a small VLM to extract high-fidelity object descriptions , which was best but slow then what i want then fine-tuned YOLOv8 model for real-time speed. Crucially, all detected object data—including semantic labels and precise spatial coordinates—are stored in a structured JSON database. This creates a "persistent memory" that the robot can query to locate objects even when they are no longer in the immediate camera frame.
The "Robot Brain": I benchmarked several models for high-level reasoning, including DeepSeek-R1:8B, Qwen2-VL:8B, and Mistral-3:3B. My testing concluded that GPT-OSS:20B no doubt delivered the most reliable reasoning for complex task planning. While performance is highly dependent on context and prompt engineering, this model proved superior in bridging the gap between natural language prompts and Nav2 low-level execuspecializedtion.
Spatial Awareness & Fusion: Object locations are precisely localized using RGB-D depth camera data. The system architecture is designed for scalability, supporting the synchronization of 2D LiDAR with 3D depth cameras to create high-fidelity, multi-modal semantic maps.
Human-Robot Interaction (HRI): To provide a seamless user experience, I integrated Nvidia Parakeet-TDT (0.6b-v2) for STT and Edge-TTS for speech synthesis. The robot’s internal reasoning is visualized through a custom PyQt-based animation interface that reflects the "thought process" in real-time.
- AI Stack: GPT-OSS (Brain), YOLOv8 (Vision),VLM, Ollama
- Speech & UI: Nvidia Parakeet-TDT, Edge-TTS, PyQt5 Animations
- Robotics: ROS2 Humble, Nav2, SLAM Toolbox
- Sensors: 2D LiDAR & RGB-D Camera (Depth-Spatial Mapping)
- Key Logic: LLM-driven task sequencing with Nav2 path planning
Demo Videos
Real-Time Semantic Mapping via YOLOv8
This phase focused on achieving low-latency perception. By integrating fine-tuned YOLOv8 with the SLAM Toolbox, I developed a pipeline that projects 2D object detections into a 3D spatial occupancy grid. This allows the robot to "see" and remember the physical location of objects in real-time without compromising navigation speed.
Advanced VLM Integration & Semantic Context
Moving beyond simple bounding boxes, I utilized a small VLM to extract rich, high-fidelity descriptions of the environment and bounding box on object. This integration allows the robot to understand visual context and identify unique object attributes—such as color, texture, and state—creating a "meaning-aware" semantic map rather than just a geometric occupancy grid.
To ensure the system remained performant for active movement, I finalized the pipeline by integrating YOLOv8. This allowed the robot to use high-speed tracking and real-time navigation updates.
Sequential Task Planning & Goal Sequencing
Demonstrating the "Robot Brain" in action: The robot receives complex, natural language commands like "Go to the Android toy, go to Mario and the Asus box." Using GPT-OSS:20b, the system deconstructs these sentences into logical navigation goals.
Contextual Reasoning
This final demo showcases the integration of Long-Term Contextual Memory with spatial reasoning. When the user asks to identify a "wild animal," the robot performs zero-shot classification to identify a toy lion.
Crucially, when the user follows up with "Which object is close to it?", the robot maintains the conversation context—understanding that "it" refers to the lion identified previously. It then queries its JSON spatial database to calculate the nearest neighbor (the yellow traffic cone) and autonomously plans a path to that coordinate via Nav2, even without the user repeating the object's name.