Intelligent control of a sensor-actuator system via kernelized least-squares policy iteration

Bo Liu, Sanfeng Chen, Shuai Li, Yongsheng Liang

Research output: Journal article publicationJournal articleAcademic researchpeer-review

3 Citations (Scopus)


In this paper a new framework, called Compressive Kernelized Reinforcement Learning (CKRL), for computing near-optimal policies in sequential decision making with uncertainty is proposed via incorporating the non-adaptive data-independent Random Projections and nonparametric Kernelized Least-squares Policy Iteration (KLSPI). Random Projections are a fast, non-adaptive dimensionality reduction framework in which high-dimensionality data is projected onto a random lower-dimension subspace via spherically random rotation and coordination sampling. KLSPI introduce kernel trick into the LSPI framework for Reinforcement Learning, often achieving faster convergence and providing automatic feature selection via various kernel sparsification approaches. In this approach, policies are computed in a low-dimensional subspace generated by projecting the high-dimensional features onto a set of random basis. We first show how Random Projections constitute an efficient sparsification technique and how our method often converges faster than regular LSPI, while at lower computational costs. Theoretical foundation underlying this approach is a fast approximation of Singular Value Decomposition (SVD). Finally, simulation results are exhibited on benchmark MDP domains, which confirm gains both in computation time and in performance in large feature spaces.
Original languageEnglish
Pages (from-to)2632-2653
Number of pages22
Issue number3
Publication statusPublished - 1 Mar 2012
Externally publishedYes


  • Kernelized least square policy iteration
  • Markov decision process
  • Random projections
  • Sensor-actuator systems

ASJC Scopus subject areas

  • Analytical Chemistry
  • Atomic and Molecular Physics, and Optics
  • Biochemistry
  • Electrical and Electronic Engineering

Cite this