Abstract
Language-conditioned manipulation (LCM) couples vision, language, and control to enable natural instruction following, making it a promising direction in embodied AI research. Researchers have combined imitation learning (IL) with reinforcement learning (RL), where IL provides sample-efficient initialization and RL refines the policy through feedback to maximize long-term returns. Despite progress, safe LCM execution faces two challenges: (1) a modality gap that disrupts semantic-spatial correspondence, degrading action quality, and (2) sparse safety costs in high-dimensional states that make constraint estimation and semantic–safety alignment difficult. In this paper, we present Semantic-Aware Behavior Optimization with Safety Reinforcement Feedback, a novel framework that mitigates multimodal misalignment through context-aware initialization and enforces semantic–safety consistency during execution. We introduce spatial–semantic priors to align language goals with spatial observations and embed safety cues. Specifically, a goal prior aligns modalities for context-aware initialization during behavioral cloning, while a constraint prior combined with a feasibility metric provides learnable supervision for safety reinforcement fine-tuning. Evaluations show our framework achieves a 6.7% relative improvement in simulation and effectively reduces safety violations in real-world tasks across diverse scenarios.
| Original language | English |
|---|---|
| Pages (from-to) | 7588 - 7595 |
| Number of pages | 8 |
| Journal | IEEE Robotics and Automation Letters |
| Volume | 11 |
| Issue number | 6 |
| DOIs | |
| Publication status | Published - 1 Jun 2026 |
Keywords
- Imitation learning
- deep learning in grasping and manipulation
- semantic scene understanding
ASJC Scopus subject areas
- Control and Systems Engineering
- Biomedical Engineering
- Human-Computer Interaction
- Mechanical Engineering
- Computer Vision and Pattern Recognition
- Computer Science Applications
- Control and Optimization
- Artificial Intelligence
Fingerprint
Dive into the research topics of 'Semantic-Aware Behavior Optimization With Safety Reinforcement Feedback for Language-Conditioned Manipulation'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver