Online internet traffic measurement and monitoring using spark streaming

Due to the explosive growth of Internet traffic, network operators must be able to monitor the whole network situations and manage their network resources in an efficient way. Traditional network analysis method that works on a single machine are no longer suitable for this huge traffic data due to its poor processing ability. Some big data frameworks, such as Hadoop and Spark, can handle such analysis job even for large network traffic, but they are inherently designed for offline data analysis. In this paper, we treat the online network analysis as a stream analysis problem and use Spark Streaming to cope with the high-speed Internet traffic data in real time. The system consists of two parts, collector and stream processor. Firstly, several collectors capture network traffic data from switches through mirrored ports and send the packet information to a central stream processor which is a cluster running Spark Streaming. Then, the stream processor analyzes the input data streams and calculates Internet performance metrics. We take TCP performance monitoring as an example to show how network measurement can be done using the stream processing platform. Finally, we conducted typical experiments in a cluster of 3 computers with the standalone mode, showing that our system performs well in huge Internet traffic measurement and monitoring.