Word segmentation method and device

Abstract

The invention provides a word segmentation method and device. The method comprises the following steps: dividing a to-be-processed corpus into a plurality of corpus segments according to a preset granularity; inserting mask segments among the plurality of corpus segments, and inputting a to-be-predicted corpus containing the plurality of corpus segments and the mask segments into a pre-training language model; corpus information in the mask fragments adjacent to the corpus fragments is predicted through a pre-training language model; and performing word segmentation processing on the to-be-processed corpus based on the plurality of corpus segments and the corpus information to obtain a target word segmentation result. According to the method, the corpus information of the mask fragment can be predicted through the pre-training language model, so that word segmentation processing is completed through the corpus information obtained through prediction, word segmentation can be completed without the help of a dictionary or a word segmentation text, efficiency reduction caused by manual construction of the dictionary or the word segmentation text is avoided, and word segmentation efficiency is improved.

Description

technical field [0001] The present disclosure relates to the technical field of data processing, and in particular, to a word segmentation method and device. Background technique [0002] With the development of artificial intelligence technology, Natural Language Processing (NLP) has become one of the important branches. In natural language processing, corpus data needs to be segmented to provide a basis for subsequent semantic recognition. [0003] At present, there are two main methods of Chinese word segmentation: one is based on the dictionary word segmentation algorithm, that is, the string to be matched is matched with an artificially constructed dictionary. If the word corresponding to the string is queried in the dictionary, it indicates a match. Success, the word is recognized. For example, forward maximum matching method, reverse maximum matching method, bidirectional matching tokenization method, etc. Another way is to use a statistical-based word segmentation...

AI Technical Summary

Problems solved by technology

In the above methods, dictionaries or word segmentation texts are usually created manually. Due to the large scale of dictionaries and word segmentation texts, more manpower is required, the efficiency is low, and the establishment and maintenance costs are high.

Images