Web page text classification algorithm research based on web page link analysis and support vector machine

Abstract

The invention discloses web page text classification algorithm research based on web page link analysis and a support vector machine and relates to the technical field of web page classification. The method includes the specific steps that 1, a large number of web pages are divided into a training set and a test set; 2, the web pages (including the training set and the test set) are preprocessed; 3, the word frequencies of feature words in each web page in the training set are calculated; 4, the weights of the feature words in each web page in the training set are calculated; 5, feature vectors of each class in the test set are calculated; 6, text feature vectors of each web page in the training set are calculated; 7, the minimum similarity value is determined as the threshold value; 8, the number of the feature words is reduced to the maximum degree; 9, text feature vectors of the web pages in the test set are classified; 10, the similarity between the classified web pages and the feature vectors is calculated and tested at the same time. A method in which a space vector model and the support vector machine is adopted is used, and the web page text classification algorithm research has the advantages of being short in classification time, high in recall rate, low in memory requirement and high in learning rate.

Description

Technical field: [0001] The invention relates to the research of webpage text classification algorithm based on webpage link analysis and support vector machine, and belongs to the technical field of webpage classification. Background technique: [0002] With the rapid development of computer and communication technology, the Internet is rapidly popularized and used, and the web pages on the network are growing at a geometric progression rate. Facing the explosive growth of massive network information, how to quickly and effectively obtain useful and interesting information is becoming more and more important. Therefore, effectively organizing and managing web resources and shortening the time for users to obtain the required information has become an urgent problem to be solved. Webpage classification technology emerged as the times require, and has gradually become a research hotspot in the field of machine learning after text classification. [0003] The traditional cla...

AI Technical Summary

Problems solved by technology

However, this method of manual classification has many disadvantages: first, when the amount of webpage texts increases rapidly, it becomes impractical to use manual classification methods to classify, and it needs to consume a lot of human resources; Text classification cannot guarantee a high classification accuracy, mainly due to the differences in subjective factors such as each person's experience and knowledge, and the classification results may be inconsistent

There are still many problems in these web page text automatic classification technologies, such as the dimensionality of the web page text feature space is too high, resulting in large storage space and slow classification speed; web pages contain a large number of noise information such as website tags and advertisements, which seriously interfere with the analysis of web pages. The determination of the category reduces the accuracy of the classification; at the same time, the information in different positions in the webpage has different abilities to express the webpage, which has a certain impact on the accuracy of the classification

Images