Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A data processing method and device

A data processing and data block technology, applied in the field of data processing, can solve the problems of persistent service processing pressure

Active Publication Date: 2019-10-01
ALIBABA GRP HLDG LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, the problem is that in the big data scenario, the speed at which processing devices read and confirm data blocks in parallel is much faster than the speed of persistence, and there will be a large number of read and confirmed data blocks waiting for the persistence service to persist them, resulting in The processing pressure of persistence service is very high

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data processing method and device
  • A data processing method and device
  • A data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0057] figure 1 A method flowchart of a data processing method provided by an embodiment of the present invention, the method includes:

[0058] S101: The processing device reads a data block from the read data block queue.

[0059] S102: The processing device generates a confirmation object having a one-to-one association relationship with the data block, and the confirmation object is used to record a confirmation state of the data block having the association relationship.

[0060] For example, optionally, the processing device may read data blocks from the read data block queue in a parallel reading manner.

[0061] Optionally, the confirmation object is used to record the confirmation status of the data block with the association relationship, specifically including:

[0062] If the data block having the association relationship with the confirmation object has not been read and confirmed, the confirmation status of the confirmation object is unconfirmed;

[0063] If t...

Embodiment 2

[0077] Use the traditional method to persist each database that has been read and confirmed. Since the speed of persistence cannot keep up with the speed of reading confirmation for parallel reading, there will be a large number of databases that have been read and confirmed waiting to be persisted. In the state, that is, in the situation of not being persisted, if the processing device reads the database interrupt at this time, when the processing device continues to read the data block from the read data block queue again , the processing device needs to re-read these massive data blocks that have been read and confirmed before, thus causing a waste of system resources and reducing the processing efficiency of the processing device.

[0078] exist figure 1 On the basis of the corresponding embodiment, image 3 It is a schematic diagram of a method for continuing reading after reading interruption provided by an embodiment of the present invention. If the processing device ...

Embodiment 3

[0086] Figure 4 A device structure diagram of a data processing device provided in an embodiment of the present invention, including:

[0087] The reading unit 401 is configured to read data blocks from the read data block queue.

[0088] For example, optionally, the reading unit 401 reads data blocks from the read data block queue in a parallel reading manner. in order to increase the reading speed.

[0089] The generating unit 402 is configured to generate a confirmation object having a one-to-one association relationship with the data block, and the confirmation object is used to record the confirmation state of the data block having the association relationship.

[0090] For example, optionally, the confirmation object is used to record the confirmation status of the data block with the association relationship, specifically including:

[0091] If the data block having the association relationship with the confirmation object has not been read and confirmed, the confir...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

Disclosed are a data processing method and apparatus. A processing device reads data blocks from a read data block queue; generates confirmation objects that are in a one-to-one association relationship with the data blocks, the confirmation objects being used for recording confirmation states of the data blocks having the association relationship; and establishes a confirmation queue comprising N confirmation objects, and executes persistent operations on the read and confirmed data blocks according to the confirmation states of the confirmation objects in the confirmation queue. Accordingly, the processing device determines, according to the confirmation states of the confirmation objects in the confirmation queue and the arrangement order of the confirmation queue, a previous data block of a data block whose confirmation state is unconfirmed and that is early read by the processing device, and it is not required to perform persistent operations on each data block that is read and confirmed, thereby reducing the processing pressure of persistent services in the case in which the data read precision is ensured.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a data processing method and device. Background technique [0002] In Internet data processing, processing devices often need to read data blocks from a database or other data storage units, and process or confirm the read data blocks. The data to be read in a data storage unit such as a database is generally in the form of one or more data block queues, and the reading device sequentially reads and confirms data blocks from the one or more data block queues. [0003] In the big data scenario, the data volume of the data block queue is very large, and the efficiency of serial reading or single-threaded reading of data blocks is too low. At present, reading devices generally adopt a method of reading data blocks in parallel to improve reading efficiency. The way of parallel reading will cause the order of reading data blocks from the data block queue to be different from the order...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/2457
Inventor 杨天洋
Owner ALIBABA GRP HLDG LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com