Unstructured data management method and device based on AI (Artificial Intelligence)

Abstract

The invention discloses an unstructured data management method and device based on AI (Artificial Intelligence). The method comprises the following steps that: through HDFS (Hadoop Distributed File System), HBase and Nosql, storing a large scale of unstructured data objects; through an AI intelligence algorithm, extracting data features from the unstructured data objects, in addition, storing theextracted data features in an external table, and constructing an unstructured data model; and using an SQL (Structured Query Language) to retrieve the feature table of the unstructured data object, and realizing the management of the unstructured data, wherein the model is used for carrying out similarity search on the unstructured data. By use of the method, the query accuracy and the expansibility of the unstructured data can be effectively improved, the core data feature value extraction of the unstructured data is realized, the unstructured data can be conveniently identified, retrieved and used, and the diversity and the flexibility of value added service are fully embodied.

Description

technical field [0001] The present invention relates to the technical field of unstructured data, in particular to an AI (Artificial Intelligence, artificial intelligence)-based unstructured data management method and device. Background technique [0002] Semantic information of unstructured data includes format information, content information, etc. However, unstructured data has a huge amount and various formats, and the content information is difficult to completely extract and store, and the storage cost is extremely high. The main methods commonly used in related technologies are: through Preset parsing rules or feature templates, and then extract keywords to obtain data information and directly store metadata information of unstructured data in the index table. [0003] However, most of the existing unstructured processing methods and devices preset corresponding templates or rules for parsing and matching keywords in unstructured data, or directly store the metadata i...

AI Technical Summary

Problems solved by technology

[0003] However, most of the existing unstructured processing methods and devices preset corresponding templates or rules for parsing and matching keywords in unstructured data, or directly store the metadata information of known data in In the index table, the potential important data information of other unstructured data is buried in the unstructured data, and the potential value of the data cannot be excavated. It is necessary to store the metadata information in advance, which undoubtedly increases the pressure on storage and Cost, due to the explosive growth of unstructured data such as images, sounds, and videos in enterprises, and unstructured data is not as easy to retrieve and utilize as structured data, making it difficult to manage and cannot be used effectively. solve

Images