Method for implementing magnanimity broadcast data warehouse-in

Abstract

The invention provides a realizing method for warehousing mass broadcast data, comprising the steps of: receiving the broadcast data according to certain period, caching the data information in a manner of data file in terms of types, time, and columns, and accomplishing data parsing and warehousing process in the way of polling, generating relevant receiving statistics information in the process of receiving, parsing and warehousing to send to a warehousing monitor client so that users can monitor the information in real time, thus achieving the stable and punctual processing to the mass broadcast data, improving processing ability to the mass data in practical application through double-machine thermal backup technology, multi-course technology, file storage technology and SQL*Loader technology, ensuring stability of application and data integrality, and enhancing performances of the application system.

Description

technical field [0001] The invention belongs to the field of computer data collection, and in particular relates to a method for realizing storage of massive broadcast data. Background technique [0002] Today, with the increasing development of networking, many businesses in the fields of finance, telecommunications and large-scale system testing are completed through local area networks, and business data are collected and broadcast through the network, and the data in these fields often belong to the scope of massive data. Typical features of data processing are: [0003] Networked transmission: All data is transmitted and exchanged through the network, with strong real-time performance. [0004] Large amount of data: The amount of data is very large, which can meet the requirements of terabytes of data. [0005] Complex data format: Since the data is transmitted in the network, the data transmission format is stipulated in the agreement. When there are multiple data ty...

AI Technical Summary

Problems solved by technology

[0009] 1. The cache size is not easy to determine, and the cache can easily lead to overflow

Regardless of whether the data queue or the shared buffer area is used, its space size is not easy to determine, and it needs to be adjusted in real time according to the operating status. Moreover, for services with a very large amount of broadcast data, it is very easy to cause buffer overflow problems, resulting in system abnormalities

[0010] 2. The system implementation is complicated and the reliability is low

Due to the complex broadcast message format and complex business processing logic in some industries, it is difficult to implement the system and the structure is complex. This complex system structure and implementation method can easily lead to poor system reliability.

[0011] 3. When the broadcasting speed is fast, the data is easily lost

When the amount of broadcast data is large and the broadcast rate is high, the system processing speed may not meet the real-time requirements of data processing, which will lead to data loss

[0012] 4. The speed of data storage is low

Although the batch storage method can increase the data storage rate, in the process of massive data processing, the batch storage method is not as fast as the SQL*Loader method.

[0013] The obvious problem with the second processing method is that a set of file system access strategies needs to be established, which does not give full play to the advantages of the database in storing massive data, and it is difficult to analyze and utilize the data again

Images