A Method of Webpage Text Extraction

Abstract

The invention provides a webpage content extracting method. The method comprises the following steps of I, preprocessing a webpage, II, searching for the longest series in the webpage, III, establishing a DOM tree and searching for the nodes corresponding to the longest series according to the DOM tree, IV, determining a beginning node and a finishing node according to labels of the nodes corresponding to the longest series, V, checking and filtering the beginning node and the finishing node, and VI, outputting text in the filtered beginning node and text in the filtered finishing node. The method overcomes the defect of a module or blocking technique in news content extraction application, searches for seed paragraphs based on the longest series and improves webpage content extracting work efficiency and accuracy.

Description

technical field [0001] The invention relates to a method in the field of computers, in particular to a method for extracting news webpage text content based on searching for "longest string" to find symbolic nodes. Background technique [0002] In the field of news (or information) search, news text extraction is an essential link, and the quality of its text extraction determines the quality of news search and user experience. [0003] At present, news text extraction methods come in various formats, which can be divided into two categories according to whether templates are used: template-based (or wrapper)-based extraction and non-template-based extraction. [0004] Template-based extraction: first define the template, and then write a program to parse and execute the template to obtain data. According to the template generation method, it can be divided into: manual template extraction and automatic template extraction. Manual template extraction. For the extracted ta...

AI Technical Summary

Problems solved by technology

[0006] The disadvantage of manually writing templates is that it takes a lot of human resources to write templates, and as the target website changes, the cost of maintaining templates is also very high

The disadvantage of the automatic template method is that the algorithm is complex, and at the same time, it also needs to periodically monitor the target website to maintain the changes of the template

Regardless of whether the template is generated manually or automatically, the assumption is that the data of the website is generated through the template. Some large websites have basically no problem, that is, different entrances may have different templates, but for many small and medium websites, the templating is not Very good, using template extraction can only extract most of the information, there are more opportunities to contain spam

Vision-based page segmentation algorithm is not suitable for news search engine applications due to complex rules and low performance

Images