Online internet traffic measurement and monitoring using spark streaming

Baojun Zhou, Jie Li, Song Guo, Jinsong Wu, Yongqiang Hu, Lihua Zhu

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review


Due to the explosive growth of Internet traffic, network operators must be able to monitor the whole network situations and manage their network resources in an efficient way. Traditional network analysis method that works on a single machine are no longer suitable for this huge traffic data due to its poor processing ability. Some big data frameworks, such as Hadoop and Spark, can handle such analysis job even for large network traffic, but they are inherently designed for offline data analysis. In this paper, we treat the online network analysis as a stream analysis problem and use Spark Streaming to cope with the high-speed Internet traffic data in real time. The system consists of two parts, collector and stream processor. Firstly, several collectors capture network traffic data from switches through mirrored ports and send the packet information to a central stream processor which is a cluster running Spark Streaming. Then, the stream processor analyzes the input data streams and calculates Internet performance metrics. We take TCP performance monitoring as an example to show how network measurement can be done using the stream processing platform. Finally, we conducted typical experiments in a cluster of 3 computers with the standalone mode, showing that our system performs well in huge Internet traffic measurement and monitoring.

Original languageEnglish
Title of host publication2017 IEEE Global Communications Conference, GLOBECOM 2017 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages6
ISBN (Electronic)9781509050192
Publication statusPublished - 10 Jan 2018
Event2017 IEEE Global Communications Conference, GLOBECOM 2017 - Singapore, Singapore
Duration: 4 Dec 20178 Dec 2017

Publication series

Name2017 IEEE Global Communications Conference, GLOBECOM 2017 - Proceedings


Conference2017 IEEE Global Communications Conference, GLOBECOM 2017

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture
  • Safety, Risk, Reliability and Quality

Cite this