Abdominal-Waving Control of Tethered Bumblebees Based on Sarsa with Transformed Reward

Nenggan Zheng, Qian Ma, Mengjie Jin, Shaomin Zhang, Nan Guan, Qiang Yang, Jianhua Dai

Research output: Journal article publicationJournal articleAcademic researchpeer-review

4 Citations (Scopus)

Abstract

Cyborg insects have attracted great attention as the flight performance they have is incomparable by micro aerial vehicles and play a critical role in supporting extensive applications. Approaches to construct cyborg insects consist of two major issues: 1) the stimulating paradigm and 2) the control policy. At present, most cyborg insects are constructed based on invasive methods, requiring the implantation of electrodes into neural or muscle systems, which would harm the insects. As the control policy is basically manual control, the shortcomings of which lie in the requirement of excessive amount of experiments and focused attention. This paper presents the design and implementation of a noninvasive and much safer cyborg insect system based on visual stimulation. The tethered paradigm is adopted here and we look at controlling the flight behavior of bumblebees, especially the abdominal-waving behavior, in the context of a model-free reinforcement learning problem. The problem is formulated as a finite and deterministic Markov decision process, where the agent is designed to change the abdominal-waving behavior from the initial state to the target state. Sarsa with transformed reward function which can speed up the learning process is employed to learn the optimal control policy. Learned policies are compared to the stochastic one by evaluating the results of ten bumblebees, demonstrating that abdominal-waving state can be modulated to approximate the target state quickly with small deviation.

Original languageEnglish
Article number8393465
Pages (from-to)3064-3073
Number of pages10
JournalIEEE Transactions on Cybernetics
Volume49
Issue number8
DOIs
Publication statusPublished - Aug 2019
Externally publishedYes

Keywords

  • Abdominal-waving
  • cyborg insect
  • reinforcement learning (RL)
  • Sarsa
  • transformed reward

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Information Systems
  • Human-Computer Interaction
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this