A scalable intelligent non-content-based spam-filtering framework

Yong Hu, Ce Guo, Wai Ting Ngai, Mei Liu, Shifeng Chen

Research output: Journal article publicationJournal articleAcademic researchpeer-review

35 Citations (Scopus)

Abstract

Designing a spam-filtering system that can run efficiently on heavily burdened servers is particularly important to the widely used email service providers (ESPs) (e.g., Hotmail, Yahoo, and Gmail) who have to deal with millions of emails everyday. Two primary challenges these companies face in spam filtering are efficiency and scalability. This study is undertaken to develop an efficient and scalable spam-filtering framework for heavily burdened email servers. We propose an Intelligent Hybrid Spam-Filtering Framework (IHSFF) to detect spam by analyzing only email headers. This framework is especially suitable for giant email servers because of its efficiency and scalability. The proposed filtering system may bedeployed alone or in conjunction with other filters. We extract five features from the email header, namely "originator field", "destination field", "X-Mailer field", "sender server IP address" and "mail subject". Email subjects are digitalized using an algorithm based on n-grams for better performance. Moreover, using real-world data from a well-known ESP in China, we employ various machine-learning algorithms to test the model. Experimental results show that the framework using the Random Forest algorithm achieves good accuracy, recall, precision, and F-measure. With the addition of MetaCost framework, the model works stably well and incurs small costs in various cost-sensitive scenarios.
Original languageEnglish
Pages (from-to)8557-8565
Number of pages9
JournalExpert Systems with Applications
Volume37
Issue number12
DOIs
Publication statusPublished - 1 Jan 2010

Keywords

  • Email header
  • Email server
  • N-Gram algorithm
  • Scalable Intelligent Hybrid Spam-Filtering

ASJC Scopus subject areas

  • Engineering(all)
  • Computer Science Applications
  • Artificial Intelligence

Cite this