Surgical manipulators are becoming more popular in modern operating theatres. Robots which work side-by-side with the surgeon and perform supportive tasks are one of the examples. However, how to allow the user to control the robot in a user-friendly manner is challenging. In this paper, we present our work on developing an image-based adaptive user interface to control a robot which assist in uterus positioning during laparoscopic hysterectomy for the hand-busy surgeon. The presented interface can be operated in two different modes, the pick and place mode, and the command specifying mode. Under the pick and place mode, the user specifies the desired starting point and ending point of the manipulation with his/her eyes and the robot drives automatically based on these points specified by the user; under the command specifying mode, the user specifies which joint and in which direction to move by looking at features of the laparoscopic monitor, then a driving command would be sent to the robot. Details of these two control approaches and the experimental results demonstrating how they work in uterus positioning are presented.