Big data statistics method and system, computer device and storage medium

Abstract

The embodiment of the invention discloses a big data statistics method and system, a computer device and a storage medium. The method provided by the embodiment of the invention comprises the following steps: reading binlog of a Mysql database, and sequentially putting log records into a message queue; The message queue is consumed through an ETL service, log records in the message queue are extracted, cleaned, converted and loaded to obtain corresponding business data, and the business data are loaded to a corresponding data warehouse; carrying out real-time analysis, aggregation, query and offline calculation on the business data through a Spark distributed query engine to obtain a corresponding statistical result; According to the technical scheme, the data is imported into the warehouse in an incremental mode, the data are cleaned and stored after being cleaned, the statistical data are calculated in advance through offline calculation, the statistical data are directly taken out when the service system is used, the statistical speed is increased, and the statistical pressure of the database is reduced.

Description

technical field [0001] The present application relates to the field of big data technology, and in particular to a big data statistical method, system, computer equipment and storage medium. Background technique [0002] At present, with the explosive growth of the Internet, the amount of data is increasing rapidly, and the loss of the database is increasing. Some statistics and other operations are likely to cause database table locking and freezing. The master-slave deployment of the database will occupy a large amount of server IO. Especially for master-slave replication with relatively large database files, the system is easily stuck. At the beginning, there were not many data services and the pressure was not great, so a database server could meet the demand. With the rapid development of business and more and more data, it is necessary to deploy services such as DB and Web Server separately. However, as the number of users continues to grow and some report statistics...

AI Technical Summary

Problems solved by technology

[0003] The embodiment of the present application provides a big data statistics method, system, computer equipment and storage medium, which are used to solve the problem in the prior art that operations such as large data volume statistics will cause the resource occupation of the database to become larger, resulting in the failure of data statistics

Images