Bejo: Behavior based job classification for resource consumption prediction in the cloud

Resource prediction (e.g. CPU/memory utilization) of cloud computing jobs has attracted substantial amount of attention. Existing works use regression methods based on historical information of jobs, with an impractical assumption that the job to be predicted has the same class as the historical jobs. To address this problem, we propose to take the category of the jobs into consideration for effective resource prediction. Existing works on job classification either ignores the temporal variance of resource consumption during job execution or use it in a naive way, resulting in unsatisfactory classification accuracy and/or slow speed. In this paper, we introduce a new and efficient job classification approach, called Bejo. Inspired by the textual document classification methods, which use distribution of text words to describe and classify a document, Bejo treats the job as a document, assigns each collected resource consumption snapshot to a certain 'resource word', and uses the distribution of the words to describe and classify a job. An l1 norm minimization formulation is used to assign each resource snapshot to a resource word, to especially address the unique challenges of high noise and tight time budget of cloud job classification. We collect a comprehensive dataset for job classification and resource consumption prediction on cloud platforms, and demonstrate superior quality and efficiency of Bejo over state-of-the-art algorithms. Experiments also show the relative error of resource consumption prediction can be dramatically reduced by adding an extra job classification step to the existing regression methods.