Data processing method and system, computer equipment and storage medium

Abstract

The invention relates to a data processing method and system, computer equipment and a storage medium. The method comprises the following steps: acquiring a target data stream through a data source; preprocessing the target data stream, and converting the target data stream into text data with a unified data exchange format; carrying out deep machine learning on the text data, then constructing anindustry knowledge graph, and carrying out distributed storage on the industry knowledge graph; and outputting a digital product according to the product requirements of the industry knowledge graph.Through the method, system reconstruction is carried out based on big data processing and artificial intelligence as support, an original business process is optimized, functional modules of the system are redivided, development of new products in data processing is redefined, and the technical characteristics of multi-backup, elastic expansion and the like of a big data distributed storage technology are utilized. The data security is ensured, and system maintenance, storage capacity expansion and performance upgrading are facilitated.

Description

technical field [0001] The present application relates to the field of big data, in particular to a data processing method, system, computer equipment and storage medium. Background technique [0002] The existing data processing platform covers reports, news writing, quality control, publishing and other links. It is an online collaborative office system with production management functions and can provide data support for the front-end website. However, the platform system has been in operation for many years. With the development of business, it can no longer meet the requirements of business development in terms of data processing capacity, user openness, data independence, etc. Therefore, a new project construction requirement for a big data processing platform is proposed. [0003] On the other hand, since the traditional digital processing system needs to collect, analyze, organize and output data, the technical knowledge required by the personnel required for each st...

AI Technical Summary

Problems solved by technology

[0002] The existing data processing platform covers reports, news writing, quality control, publishing and other links. It is an online collaborative office system with production management functions and can provide data support for the front-end website. However, the platform system has been in operation for many years. With the development of business, it can no longer meet the requirements of business development in terms of data processing capabilities, user openness, and data independence. Therefore, a new project construction requirement for a big data processing platform is proposed

[0003] On the other hand, since the traditional digital processing system needs to collect, analyze, organize and output data, the technical knowledge required by the personnel required for each step of the business process is quite different, and the data analysis-related business personnel need to have higher Technical know-how, which leads to high costs for the entire data processing process

For example, when dealing with data in current content-based industries such as the media industry and industrial research industry, due to the different data formats and standards, it is impossible to make comprehensive use of the data, and the data has not exerted greater value.

In addition, due to its non-distributed architecture, the traditional storage technology cannot add servers, and the server needs to be shut down to upgrade the server when processing massive data. Therefore, the traditional data processing system has insufficient performance for the storage and reading of massive big data. Fast and effective processing of massive data

Images