Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data processing method, device and system

A data processing and data technology, applied in the field of data processing, can solve the problems of storage space waste, data redundancy, etc., and achieve the effect of avoiding waste, reducing the moving distance of the magnetic head, and improving the performance of query statistics

Inactive Publication Date: 2013-09-11
上海淼云文化传播有限公司
View PDF0 Cites 26 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to provide a data processing method, device and system to solve the technical problem that in the prior art it is impossible to implement data deduplication while data storage, avoiding data redundancy and waste of storage space

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, device and system
  • Data processing method, device and system
  • Data processing method, device and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0083] refer to image 3 , which shows a flow chart of Embodiment 2 of a data processing method provided by the present invention. Embodiment 2 of the method of the present invention is based on Embodiment 1 of the method, and further includes the following steps:

[0084] Step 301: Receive a data query request, where the data query request includes query conditions.

[0085] Wherein, after the data query request input by the user is received, the query condition included in the data query request is parsed.

[0086] Step 302: Perform hash algorithm calculation on the query condition to obtain the query key value.

[0087] Wherein, it should be noted that, when calculating the hash algorithm for the query condition, the hash algorithm described in the first method embodiment of the present invention needs to be calculated.

[0088] Step 303: In the key values ​​of the data set, check whether there is a key value matching the query key value, if yes, perform step 304, otherwi...

Embodiment 2

[0146] refer to Figure 9 , which shows a schematic structural diagram of Embodiment 4 of a data processing device provided by the present invention. Based on Embodiment 2 of the device of the present invention, the device may further include:

[0147] The data cache unit 609 is configured to store the data returned by the data search unit 602 into a preset cache data set.

[0148] Wherein, the data search unit 602 stores the returned data as cache data in the cache data set after completing a successful data query. When the receiving unit 605 receives the data query request again, the data cache unit 609 first judges whether the cached data set contains the data corresponding to the data query request, and if so, the data The data requested by the query has been queried, then the data search unit 602 can directly perform data query in the cached data set, if not, it means that the data requested by the data query has not been queried, then the The data search unit 602 perfo...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data processing method, device and system. The data processing method comprises the steps that to-be-stored data is subjected to hash algorithm calculation to obtain a key value of the to-be-stored data; and whether a key value matched with the key value is contained is searched in a preset data set, if yes, the to-be-stored data is rejected, otherwise, the to-be-stored data is stored in a column type storage method, and the key value of the to-be-stored data is stored in the data set. According to the data processing method, device and system provided by the embodiment of the invention, the hash calculation is performed before storage of the to-be-stored data in the column type storage method, duplication eliminating processing is performed on the to-be-stored data based on the obtained key value, so that data redundancy during mass data processing is avoided, and storage space waste is avoided.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a data processing method, device and system. Background technique [0002] Massive data refers to fields such as enterprise-level software, the Internet, and cloud computing, which need to process large-capacity data. At present, the mass data processing scheme is generally adopted as a slice separation scheme. [0003] The above slicing separation scheme means that when storing data, the data to be stored is stored in slices according to the slicing rules, that is, the data to be stored is stored in different data slices according to the data attributes, and the horizontal expansion of data storage is realized. to efficiently process massive amounts of data. [0004] Although the slice separation scheme can store data into different data slices, the data in the data slices is not deduplicated during the storage process, resulting in data redundancy. Moreover, as the amount of d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
Inventor 李晨马向晖
Owner 上海淼云文化传播有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com