Extraction method and retrieval method of customs data product words

Abstract

The invention provides an extraction method and a retrieval method of customs data product words. The extraction method comprises the following steps: firstly, cleaning up redundant parts in a customsdescription text, and converting the redundant parts into a better processing form; then, heuristically finding out segmentation words in the customs description text, and segmenting product words and description parts; replacing quantifiers and date regularities in the text with space characters or deleting the quantifiers and date regularities; deleting a description part in the text through agrammatical rule, or extracting product word groups from the data by using mutual information and left and right information entropies to obtain product word groups of which the number of words is less than or equal to 5, and adding the product word groups into a lexicon. The retrieval method comprises the steps that firstly, word segmentation is conducted on a text to be retrieved, and then retrieval is conducted in a constructed word bank through a bit map or hash map structure. According to the method, the grammatical structure, the mutual information, the character information and the specific structure information of the customs data are combined, the advantages of various information can be fully combined, and the product words can be accurately extracted and retrieved.

Description

technical field [0001] The invention belongs to the technical field of communication, and in particular relates to a method for extracting and retrieving customs data product words. Background technique [0002] Customs data and courier data generally must contain descriptions of the goods being shipped. These descriptions are descriptions of specific products, with a large amount of product word information, such as product names, product attributes, manufacturer information, product functions, and advertising words, etc. . Therefore, it is more feasible to clean and utilize product words. When we have a complete product thesaurus, we can use it to quickly retrieve existing text data, increasing the utilization rate and retrieval efficiency of text data. However, since this kind of product word contains a large number of unstructured words and sentences, it is difficult to extract its keywords, and it is difficult to have a suitable algorithm to effectively extract it. ...

AI Technical Summary

Problems solved by technology

However, since this kind of product word contains a large number of unstructured words and sentences, it is difficult to extract its keywords, and it is difficult to have a suitable algorithm to effectively extract them

[0003] The traditional product thesaurus is usually based on manual entry and network collection, which requires a lot of cost to maintain, and does not guarantee real-time performance

At present, there are still the following problems in the extraction of product words: 1. Wrong words are easy to exist in handwritten words during manual entry; 2. There are errors in the segmentation between words and sentences; 3. Product words often contain the performance of the product , quality, etc., such description sentences contain acronyms, numbers, stop words, symbols, etc., which may easily cause poor results from word segmentation

Images