Smartphones have become a necessity in people’s daily lives, and changed the way of communication at any time and place. Nowadays, mobile devices especially smartphones have to store and process a large amount of sensitive information, i.e., from personal to financial and professional data. For this reason, there is an increasing need to protect the devices from unauthorized access. In comparison with the traditional textual password, behavioral authentication can verify current users in a continuous way, which can complement the existing authentication mechanisms. With the advanced capability provided by current smartphones, users can perform various touch actions to interact with their devices. In this work, we focus on swipe behavior and aim to design a machine learning-based unlock scheme called SwipeVLock, which verifies users based on their way of swiping the phone screen with a background image. In the evaluation, we measure several typical supervised learning algorithms and conduct a user study with 30 participants. Our experimental results indicate that participants could perform well with SwipeVLock, i.e., with a success rate of 98% in the best case.