Machine-learning-based online distributed denial-of-service attack detection using spark streaming

Baojun Zhou, Jie Li, Jinsong Wu, Song Guo, Yu Gu, Zhetao Li

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

13 Citations (Scopus)


In order to cope with the increasing number of cyber attacks, network operators must monitor the whole network situations in real time. Traditional network monitoring method that usually works on a single machine, however, is no longer suitable for the huge traffic data nowadays due to its poor processing ability. In this paper, we propose a machine-learning based online Internet traffic monitoring system using Spark Streaming, a stream- processing-based big data framework, to detect DDoS attacks in real time. The system consists of three parts, collector, messaging system and stream processor. We use a correlation-based feature selection method and choose 4 most necessary network features in our machine- learning-based DDoS detection algorithm. We verify the result of feature selection method by a comparative experiment and compare the detection accuracy of 3 machine learning methods - Naive Bayes, Logistic Regression and Decision Tree. Finally, we conduct experiments in a cluster with the standalone mode, showing that our system can detect 3 typical DDoS attacks - TCP flooding, UDP flooding and ICMP flooding at the accuracy of more than 99.3%. It also shows the system performs well even for large Internet traffic.

Original languageEnglish
Title of host publication2018 IEEE International Conference on Communications, ICC 2018 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Print)9781538631805
Publication statusPublished - 27 Jul 2018
Event2018 IEEE International Conference on Communications, ICC 2018 - Kansas City, United States
Duration: 20 May 201824 May 2018

Publication series

NameIEEE International Conference on Communications
ISSN (Print)1550-3607


Conference2018 IEEE International Conference on Communications, ICC 2018
Country/TerritoryUnited States
CityKansas City

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Cite this