Method and device for real-time summarization and interval summarization of data

Abstract

The invention discloses a method and a device for real-time summarization and interval summarization of data, and relates to the technical field of computers. One specific embodiment of the method comprises the following steps that a predetermined number of RegionServers based on HBase are deployed through an HDFS (Hadoop Distributed File System); and the RegionServers respectively carry out data summarization in parallel according to predetermined fields, and the summarized data of the RegionServers are sent to the client side for carrying out secondary data summarization. According to the embodiment, complete real-time processing and storage are reduced; storage and computing resources are placed on the same machine, and data network transmission is minimized, so that the summarization time is shortened; the resource expansion without perception of the program is realized. Support for the SQL is provided, so that the readability is better; and two modes of real-time summarization and interval summarization are provided to solve scenes with different data volumes.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for real-time data collection and interval collection. Background technique [0002] Under the current background of a large amount of business data, the association and aggregation of massive real-time data has always been a difficult problem in the field. The main steps of the solution currently adopted are: aggregate the real-time data generated by each business table into Kafka's Topic; use a consumer application group, and associate the two business table programs by caching or reading back the associated fields through ElasticSearch Then store it in the wide table created by ElasticSearch; periodically start the summary application group to read the wide table data in the time period to be summarized, perform summary processing and store it in the cache or relational database. [0003] In the course of realizing the present invention, the inventor fin...

AI Technical Summary

Problems solved by technology

[0004] -ElasticSearch only stores a small amount of data in the time interval between quasi-real-time storage and data loss when the stage is summarized;

[0005] - The independent deployment of consumer application groups and summary application groups takes up a lot of machine resources, which increases the complexity and risk points of the entire system;

[0006] - Calculation and data are on different machines. Due to the huge amount of data, a large amount of network transmission is required, which leads to the occupation of a large amount of network resources in the stage summary stage, and also increases the time consumption of summary;

[0007] - Poor scalability of both computing resources and storage resources; and

[0008] -ElasticSearch API coding is poorly readable and difficult to get started

Images