Real-time stream data processing method and system for smart city

Abstract

The invention discloses a real-time stream data processing method and system of a smart city. The method comprises the steps of data collection, data type classification, data cleaning, data calculation and analysis, data display, data persistence storage and the like. The invention sets up a data collection module in the mode of mainly using Flume and supplemented by API collection, and utilizeschannel to carry out data persistence, so as to ensure reliability in the process of data transmission. Using Kafka as a stream processing platform and introducing the fault-tolerant mechanism of Flink to consume Kafka data in high level, the real real-time stream data processing can be realized, which can support millisecond-level computation. In addition, perfect window support of Flink providesmore perfect window aggregation computation API than Storm. Finally, WebSocket is used to establish the connection channel between a server and a Web end to provide real-time data service, reduce thepressure of data transmission, and improve the high concurrent access and availability of the system.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method and system for processing real-time stream data of a smart city. Background technique [0002] Among the existing smart city big data solutions, the batch processing system based on government business system data is the mainstream, and Logstash is often used as the collection tool, and Spark Streaming or Storm is used as the streaming computing framework, and finally the computing results are sent through the HTTP protocol. Send it to the web front end, and the web front end will display the page. Among them, Logstash can realize data transmission, format processing, formatted output, and powerful plug-in functions, which are often used for log processing. [0003] With the development of technology and the popularization of applications, the amount of data in smart cities is increasing day by day. The following problems will arise in the adoption of the above ...

AI Technical Summary

Problems solved by technology

[0004] 1. Data analysis is only based on the data of the government business system, the data source is single, the social reality reflected by the data lacks a three-dimensional sense, and the purpose of using big data technology to promote the development of smart cities cannot be realized;

[0005] 2. The throughput of a large amount of data will reduce the analysis ability of the Logstash collection module, consume large resources, slow the transmission speed, and there is a possibility of data loss;

[0006] 3. As a streaming computing framework, Spark Streaming is actually a "micro-batch" processing form, which cannot form a real streaming computing, and the real-time performance of data will be slowly reduced, which will cause problems such as data delay and data transmission cannot be synchronized ;

[0007] 4. HTTP is a non-persistent protocol, and the Ajax polling method needs to continuously establish connections to obtain data, and needs to continuously call server resources. The server faces the danger of high concurrency and consumes a lot of resources, and the real-time performance is relatively poor.

Images