A Large-Scale Network Streaming Data Cache Writing Method

Abstract

The invention provides a large-scale network streaming data cache-write method including that a client side packages collected data and sent the same to a server side; the server side analyzes the received data to determine data resources and unifies data formats; the data with uniform formats are written into cache regions of different inner memories according the data resources; different cache strategies are made according to the data which are written into the cache regions of the different inner memories, and the data meeting strategy trigger conditions are written into a local file system from the inner memories; the data written into the local file system are loaded to a distributed file system or distributed database according to the upper application demand; and to the data which are loaded to the distributed file system or distributed database, small data blocks are combined into large data blocks at fixed time. The large-scale network streaming data cache-write method is applicable to write of large-scale network data that are of different resources and different inflow velocity by the aid of a multi-stage cache mechanism.

Description

technical field [0001] The invention relates to the field of network technology, in particular to a method for writing large-scale network streaming data buffers. Background technique [0002] In recent years, with the rapid development of the Internet, the rapid growth of data has become a common opportunity and challenge for many industries. In today's network environment, a large number of data sources are real-time and uninterrupted, and the response time to users is also required to be real-time. These data are collected, calculated and queried in the form of streaming. For example, the network anomaly detection system collects data such as network packets and network logs for analysis, and ensures that the analysis results are returned within a certain time range to ensure high availability of the network. The characteristics of this kind of system are: there are all kinds of massive network data flowing into the system at every moment, the inflow speed is different,...

AI Technical Summary

Problems solved by technology

[0004] However, the batch processing mode of the Hadoop architecture cannot meet the requirements of real-time computing, and the system processing speed is slowed down, which is not suitable for direct data inflow; both Hbase and Hive are distributed databases, but the data formats flowing into the system from different data sources on the network are different. It cannot be directly provided to distributed databases without processing; the problem with Yahoo’s S4 and Twitter’s Storm, which are specially designed for distributed streaming platforms, is that they provide more computing power for streaming data, and all arriving After processing, the data is directly entered into the memory for calculation, and the inflowing data is not stored persistently, which cannot meet the needs of the application.

Images