Webpage uniform resource locator (URL) classification and identification method and device

Abstract

The invention discloses a webpage uniform resource locator (URL) classification and identification method and device used for improving URL classification accuracy and depth and improving efficiency under the premise that data amount required by URL classification is reduced as much as possible. The method includes analyzing a plurality of specific URLs with the identical domain name, determining a main path and sub paths contained by the URLs, building a directory tree according to the determined main path and sub paths as the directory of each stage, dividing each URL into the corresponding directories in the directory tree according to all paths contained in the URLs, determining keywords in webpages that the URLs correspond to aiming at the directory of any stage in the directory tree, and determining the class of a first keyword to be the class of the URLs under the directory when determining that the ratio of the number of the webpages with identical first keyword and the total number of the webpages in the directory is higher than the set threshold.

Description

technical field [0001] The present invention relates to the field of network technology, in particular to a method and device for classifying and identifying web page URL addresses. Background technique [0002] With the rapid development of the Internet, the amount of network data has increased sharply. Facing the huge amount of webpage information resources, it is necessary to classify and organize the huge amount of webpage information. [0003] At present, classifying webpages according to their Uniform Resource Locators (URLs) is one of the more common webpage classification methods. [0004] Traditional URL classification identification mainly has the following methods: [0005] The first method is to use web crawler technology to crawl the content of the webpage according to the URL address after obtaining the URL address of the webpage, and determine the set number of keywords (Keywords ), and determine the category to which the webpage belongs according to the det...

AI Technical Summary

Problems solved by technology

However, the matching depth of domain name matching is low. Generally, it can intelligently identify which website the URL URL belongs to. It is impossible to determine which channel or category the URL belongs to.

[0008] The fourth is to match the URL address of the obtained web page with the preset directory. If the match is successful, attribute the URL address to the category corresponding to the directory. This method can ensure the accuracy of the match, but the preset directory requires manual work. After analyzing and combing a large number of URL addresses, the workload of maintaining the directory is heavy, and it is difficult to ensure the matching coverage by manually combing and obtaining the directory.

Images