Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Distributed data management system

A distributed data and management system technology, applied in the field of distributed data management systems, can solve the problems of artificial difficulty in supporting data screening and collection, obtaining data information, huge data volume, etc., and achieve the goal of improving data utilization and accuracy Effect

Pending Publication Date: 2021-01-15
镇江睿知信息科技有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] In the current Internet era, the amount of data is increasing rapidly, and a large amount of valuable data information is inundated. It is difficult for users to obtain the data information they need in a timely, effective and low-cost manner. At the same time, the huge amount of data makes it difficult for humans to support data screening and collection.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data management system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0016] In order to make the technical solution of the present invention clearer, the present invention will be further described below in conjunction with the examples, any technical features of the technical solution of the present invention are equivalently replaced and obtained by conventional reasoning, all fall within the protection scope of the present invention.

[0017] A distributed data management system is characterized in that it includes a URL dynamic allocation module, an anti-climbing policy scheduling module, a data mark analysis module, and a data formatting module.

[0018] The URL dynamic deployment module generates a URL seed library through data configuration information;

[0019] The anti-crawling strategy scheduling module realizes four anti-crawling confrontation functions;

[0020] The data mark parsing module is responsible for parsing the information path and marking web data;

[0021] The data formatting module is responsible for storing the proces...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed data management system. The system comprises a URL dynamic allocation module, an anti-crawling strategy scheduling module, a data mark analysis module and a dataformatting module. The research and development of the system can efficiently obtain Internet hotspot information, and provide timely and refined data for users according to differentiated and personalized requirements. According to the system, a framework combining Storm real-time stream processing, database middleware, Hadoop batch processing and ELK data visualization is adopted, so that the compatibility, fault tolerance and expandability of the system are improved, and the adaptability of the system to differentiated requirements is improved.

Description

technical field [0001] The invention relates to the technical field of computers, in particular to a distributed data management system. Background technique [0002] In the current Internet era, the amount of data is increasing rapidly, and a large amount of valuable data information is inundated. It is difficult for users to obtain the data information they need in a timely, effective and low-cost manner. At the same time, the huge amount of data makes it difficult for humans to support data screening and collection. Therefore, with the development of the times, it is difficult to meet users' needs for data timeliness by relying on offline data sets and public free data sets. Therefore, an online data management system that can provide high-quality and timely data has become an indispensable part. Contents of the invention [0003] The invention provides a distributed data management system, which is characterized in that it includes a URL dynamic deployment module, an a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/955G06F16/958H04L29/06
CPCG06F16/9566G06F16/958H04L63/145
Inventor 徐雷
Owner 镇江睿知信息科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com