Data processing method, system and device based on web topic crawler

A topic crawler and data processing technology, applied in the field of data processing, can solve the problem of not being able to efficiently filter and identify information on specific topics at the same time, and achieve the effect of flexible query and improved content accuracy

Pending Publication Date: 2021-04-13
BIGO TECH PTE LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In order to crawl information on the Internet, it is necessary to use web crawler technology. The existing technical solutions focus on the crawling and structured storage of meta-information; or focus on the quantity of information obtained, which cannot meet the requirements of efficient screening and screening of specific topics at the same time. information needs

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, system and device based on web topic crawler
  • Data processing method, system and device based on web topic crawler
  • Data processing method, system and device based on web topic crawler

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] In order to make the purpose, technical solution and advantages of the present application clearer, specific embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only parts relevant to the present application are shown in the drawings but not all content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe various operations (or steps) as sequential processing, many of the operations may be performed in parallel, concurrently, or simultaneously. In addition, the order of operations can be rearranged. The proc...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a data processing method, system and device based on a web topic crawler. The method comprises the steps of receiving a crawling task and a screening task created by a user; sending the crawling task to a topic crawler center to enable the topic crawler center to obtain crawling content; and sending the screening task to a screening center to enable the screening center to obtain the target content. According to the technical scheme, the crawling task is limited by configuring the independent limiting condition, the screening content is limited by independently configuring the screening condition, the content accuracy of the topic crawler is greatly improved, and the crawling task and the screening task are separately configured, the crawling task and a screening task are executed through different functional modules respectively, so that different functional layers can effectively exert respective characteristics, crawling and use of data by crawlers are separated, more accurate content is screened out, and the screened content can be pushed for flexible query of users.

Description

technical field [0001] The embodiments of the present application relate to the technical field of data processing, and in particular, to a data processing method based on a web theme crawler, a data processing system based on a web theme crawler, a data processing device based on a web theme crawler, and a web theme crawler-based Data processing equipment and storage media. Background technique [0002] In the information age, the amount of information on the Internet is increasing exponentially, but the amount of information received by individuals and the ability to process information are limited. When learning information in a certain field, it often takes a lot of time to screen and identify information on a specific topic. Therefore, automatic collection of information in specific fields and efficient screening of information came into being. [0003] A web crawler is a program that automatically grabs Internet information according to certain rules. It searches for...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/951G06F16/9535
CPCG06F16/951G06F16/9535
Inventor 蔡开顶
Owner BIGO TECH PTE LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap