Machine writing system based on natural language processing technology

Abstract

The invention discloses a machine writing system based on a natural language processing technology, which comprises a preprocessing module, a data loading module, a language model and a main module; the preprocessing module is mainly used for preprocessing text data and comprises a vocabulary segmentation process and a vocabulary numbering process; the data loading module is mainly used for further packaging the preprocessed text data and comprises a data type conversion step and a batch data generation step; the packaged data of each batch can be input into a language model for training; the language model is mainly used for learning language rules in the text data and generating codes of a next prediction word; the main work of the main module is to integrate the functions of all the modules, design a loss function and an optimization method and complete the training work of the model. The system has the advantages of few used technologies, simplicity, high specificity, low maintenance cost and the like.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a machine writing system based on natural language processing technology. Background technique [0002] Since DeepMind's excellent work AlphaGo defeated South Korean professional Go player Lee Sedol with a huge advantage of 4:1 in 2017, the term artificial intelligence has rapidly become popular all over the world. For a time, various achievements relying on artificial intelligence technology continued to emerge, such as image recognition, speech recognition, machine writing, etc.; so that 2017 was called "the first year of AI (artificial intelligence)" by the Wall Street Journal, Forbes and Fortune magazine, Artificial intelligence is getting closer and closer to our life, and I believe it will bring more convenience to our life in the future. [0003] Machine writing, as the name suggests, is to make machines write articles like humans, which involves many ...

AI Technical Summary

Problems solved by technology

[0003] Machine writing, as the name suggests, is to make machines write articles like humans, which involves many technical problems, such as how the machine learns to recognize characters, how to judge the emotion of a sentence, how to write good articles that conform to human cognitive habits, etc.; The representative one is Baidu intelligent writing robot, which is realized based on technologies such as big data analysis, content understanding and natural language generation. Its basic creation process is mainly divided into data collection, data analysis, automatic manuscript writing, review and issuance, etc. ; Among them, the automatic manuscript writing part usually includes three stages: document planning, onlooker planning and surface realization, which respectively solve the problems of what to write, how to write and how to polish and present the manuscript; Many and complex technologies, weak specificity, high maintenance costs, etc.

Images