Collection method for vertical data of web spider

Abstract

The invention discloses a collection method for vertical data of a web spider. The collection method comprises the following steps: 1) a client establishes a URL (uniform resource locator) class library and a Page class library; 2) a client process is in connection with a server process; 3) the client generates a request message body and sends the request message body to a server; 4) the client acquires webpage header information and webpage body information; and 5) the client analyzes the webpage header information and saves the satisfactory webpage body information so as to complete the collection of the webpage data. The collection method for the vertical data of the web spider disclosed by the invention can provide more accurate information to a user and better meet the search requirements of the user, and has an accurate and stable algorithm without causing depletion of local resources, so that the collection method has a good application prospect.

Description

technical field [0001] The invention relates to the technical field of information analysis and capture, in particular to a vertical data collection method for web spiders. Background technique [0002] At present, with the rapid expansion of network information, the amount of information on the Internet is increasing. Now some search engines such as Google, Baidu, etc., etc. are free and open to all users, and they all strive to do their best in the search results. However, the above-mentioned search engines are not specially designed to search for information in a specific field. However, people often need to use search engines to help them find information in a specific field in a large amount of information. For example, job seekers search for "Suzhou Tourism" in Baidu. , there will be some travel information, but many of the travel information in the results have expired, and some even have job information from a year ago or even longer. For very time-sensitive informat...

AI Technical Summary

Problems solved by technology

[0002] At present, with the rapid expansion of network information, the amount of information on the Internet is increasing. Now some search engines such as Google, Baidu, etc., etc. are free and open to all users, and they all strive to do their best in the search results. However, the above-mentioned search engines are not specially designed to search for information in a specific field. However, people often need to use search engines to help them find information in a specific field in a large amount of information. For example, job seekers search for "Suzhou Tourism" in Baidu. , there will be some travel information, but many of the travel information in the results have expired, and some even have job information from a year ago or even longer. For very time-sensitive information such as travel information, search engines such as Baidu obviously Can not meet user needs

[0003] Also, the background of developing web search is because there are too many web pages on the Internet, and users cannot find the web pages they want. There are two problems. First: users want to find something they want from open web search. It takes a lot of time to search for information in a specific field. For example, for job hunting, if a user enters "java development" in Google, the results are all about java development skills. If you want to see information about "java development" "Positions need to be constantly turned over; the second reason for vertical search is that the number of domain-specific websites is increasing day by day. Taking recruitment as an example, there are now hundreds of domestic recruitment websites, except for 51job and Chinahr. In addition to such portals as Zhaopin and Zhaopin, each region also has its own recruitment portals. Therefore, if users want to get comprehensive recruitment information, they need to open one website after another, which is time-consuming and labor-intensive.

Images