Skip to main navigation Skip to search Skip to main content

Semantic-Aware Behavior Optimization With Safety Reinforcement Feedback for Language-Conditioned Manipulation

  • Xiuxiu Qi
  • , Jiannong Cao
  • , Julie A. McCann
  • , Chongshan Fan
  • , Hongpeng Wang

Research output: Journal article publication › Journal article › Academic research › peer-review

Abstract

Language-conditioned manipulation (LCM) couples vision, language, and control to enable natural instruction following, making it a promising direction in embodied AI research. Researchers have combined imitation learning (IL) with reinforcement learning (RL), where IL provides sample-efficient initialization and RL refines the policy through feedback to maximize long-term returns. Despite progress, safe LCM execution faces two challenges: (1) a modality gap that disrupts semantic-spatial correspondence, degrading action quality, and (2) sparse safety costs in high-dimensional states that make constraint estimation and semantic–safety alignment difficult. In this paper, we present Semantic-Aware Behavior Optimization with Safety Reinforcement Feedback, a novel framework that mitigates multimodal misalignment through context-aware initialization and enforces semantic–safety consistency during execution. We introduce spatial–semantic priors to align language goals with spatial observations and embed safety cues. Specifically, a goal prior aligns modalities for context-aware initialization during behavioral cloning, while a constraint prior combined with a feasibility metric provides learnable supervision for safety reinforcement fine-tuning. Evaluations show our framework achieves a 6.7% relative improvement in simulation and effectively reduces safety violations in real-world tasks across diverse scenarios.
Original languageEnglish
Pages (from-to)7588 - 7595
Number of pages8
JournalIEEE Robotics and Automation Letters
Volume11
Issue number6
DOIs
Publication statusPublished - 1 Jun 2026

Keywords

  • Imitation learning
  • deep learning in grasping and manipulation
  • semantic scene understanding

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Biomedical Engineering
  • Human-Computer Interaction
  • Mechanical Engineering
  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Control and Optimization
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Semantic-Aware Behavior Optimization With Safety Reinforcement Feedback for Language-Conditioned Manipulation'. Together they form a unique fingerprint.

Cite this