Method for extracting core content of webpage based on text-tag density

Abstract

This invention relates to a method for extracting a core content of a webpage based on text-tag density. The method comprises four steps of preprocessing webpage source code, estimating the range of the core content of the webpage, determining the boundary of the core content and deleting residual tags. In the step of preprocessing webpage source code, extract core elements of title, summary, description, and so on from an original webpage text, and delete the tags unrelated to the core content of the webpage in the original webpage text so as to acquire a pending text. In the step of estimating the range of the core content of the webpage, determine a general range of the core content of the webpage. In the step of determining the boundary of the core content, separately determine precise start and stop positions of the core content of the webpage text. In the step of deleting residual tags, take out the core content part and delete residual tags to acquire the core content of the webpage, which is convenient to be analyzed and processed. By adoption of this method, the DOM (Document Object Model) structure of a webpage document is unnecessary to be analyzed; the theme and the content of the webpage are not limited; the processing procedure has linear complexity; and this method is applicable to the technical applications of extracting the core contents of various kinds of webpages, denoising webpages, and so on.

Description

technical field [0001] The invention relates to the technical field of the Internet in the field of communication, in particular to a method for extracting the core content of a webpage text with linear complexity based on text-label density. Background technique [0002] With the rapid development of the Internet, the World Wide Web (WWW) has become the largest Internet database in the world. Therefore, how to effectively extract information from the World Wide Web has become a new research direction. These involve collecting, processing, and extracting information from web pages at high speed. [0003] However, in reality, in addition to the text content related to the topic, there will be a lot of irrelevant information on the web page. This content includes everything from logos, advertisements, images, navigation, sidebars, and more. Although this information can play a role in assisting browsing for web browsers, it is useless in most cases for many Internet applica...

AI Technical Summary

Problems solved by technology

Since the template largely depends on the specific structure of the web page, once the structure of the web page changes, it needs to be reset and learned, and the versatility is not strong

Images