Data processing method, system and device and readable storage medium

Abstract

The invention discloses a data processing method. The method comprises the steps of determining a to-be-processed data name and a corresponding processing operation according to an input data processing request; determining an ES field name corresponding to the name of the data to be processed according to a preset field mapping relation table; and searching ES data corresponding to the ES field name in an ES database, and executing a processing operation to process the ES data. According to the method, system and device, the problem of default field upper limit existing in the ES database is solved, the implementation is simpler, the mapping between the name of the data to be processed and the name of the ES field is performed before the ES data processing operation, the ES retrieval performance is not reduced, the response speed of service processing is not influenced, and the effect of improving the data processing efficiency of the ES database on the basis of breaking through the limitation of the ES database is achieved. The invention also provides a data processing system and device and a readable storage medium, which have the above beneficial effects.

Description

technical field [0001] The present application relates to the field of data processing, in particular to a data processing method, system, device and readable storage medium. Background technique [0002] A single relational database has horizontal and vertical tables, poor retrieval capabilities, and a bottleneck for the storage of large amounts of data. However, nosql’s support for transactions is not friendly enough, so ElasticSearch (ES for short) database is generally used for data storage and retrieval. ES is a document-oriented database. A piece of data is a document here. JSON is used as the document serialization format, which can be dynamically The fields in the extended document store a large amount of data and have excellent retrieval performance. [0003] But the disadvantage of ES database is the upper limit of 1000 default fields. Existing solutions can be roughly divided into the following three types: [0004] 1) You can expand the upper limit of document...

AI Technical Summary

Problems solved by technology

[0004] 1) You can expand the upper limit of the document field through configuration, but this solution will reduce the efficiency of ElasticSearch retrieval

[0005] 2) Communicate with business parties and users to delete uncommonly used fields, but the C-end system faces the market and cannot control the use of users

[0006] 3) Set up multi-dimensional storage and computing (such as hive). However, this solution involves code refactoring, data migration, etc., and the transformation cost is high, and the data processing efficiency is low

Images