Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data processing method, device and system in distributed storage system

A distributed storage and data processing technology, applied in the direction of electronic digital data processing, special data processing applications, file systems, etc., can solve problems such as increasing the complexity of user program processes, and achieve the effect of increasing complexity

Active Publication Date: 2020-03-10
ALIBABA GRP HLDG LTD
View PDF3 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] Embodiments of the present invention provide a data processing method, device, and system in a distributed storage system to at least solve the problem of increasing the complexity of the user program flow due to the need for the user to process the event that reads the end of the file and reopen the file to obtain the input stream. sexual technical issues

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, device and system in distributed storage system
  • Data processing method, device and system in distributed storage system
  • Data processing method, device and system in distributed storage system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0036] According to an embodiment of the present application, an embodiment of a data processing method in a distributed storage system is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be implemented in a computer system such as a set of computer-executable instructions. and, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that herein.

[0037] The method embodiment provided in Embodiment 1 of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method in a distributed storage system is shown. like figure 1 As shown, computer terminal A (or mobile device A) may include one or more processors 102 (illustrated by 102a, 102b, . or a processing device su...

Embodiment 2

[0093] According to an embodiment of the present application, a data processing apparatus in a distributed storage system for implementing the data processing method in the distributed storage system is also provided, such as Image 6 As shown, the apparatus 60 includes: a generating module 601 , an acquiring module 603 and a reading module 605 .

[0094] Wherein, the generating module 601 is used to generate a data reading request, wherein the parameters of the data reading request carry the offset and length of the read data; the acquiring module 603 is used to obtain data from the distributed storage system according to the data reading request The metadata of the file is obtained from the cache of the output stream of the file in the file, wherein the metadata of the file includes at least the current length of the file; the reading module 605 is used to read data from the file according to the current length of the obtained file .

[0095] Here, it should be noted that t...

Embodiment 3

[0110] According to an embodiment of the present application, a data processing system in a distributed storage system is also provided, and the system can execute a data processing method in the distributed storage system. Specifically, the system includes: a processor and a memory.

[0111] The memory is connected to the processor, and is used to provide the processor with instructions for processing the following processing steps: generating a data read request, wherein the parameters of the data read request carry the offset and length of the read data; according to the data read The fetch request obtains the metadata of the file from the cache of the output stream of the file in the distributed storage system, wherein the metadata of the file includes at least the current length of the file; according to the obtained current length of the file, read data from the file .

[0112] In an optional solution, when the sum of the offset and length of the read data is greater th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data processing method, device and system in a distributed storage system. The method comprises the steps that a data reading request is generated, and parameters of the datareading request carry offset and length of read data; according to the data reading request, metadata of the file are obtained from a cache of an output stream of the file in the distributed storagesystem, and the metadata of the file at least comprise the current length of the file; and reading data from the file according to the obtained current length of the file. The technical problem that the complexity of a user program process is increased due to the fact that a user needs to process an event of reading the end of a file and reopen the file to obtain an input stream is solved.

Description

technical field [0001] The present application relates to the field of computer storage, and in particular, to a data processing method, device and system in a distributed storage system. Background technique [0002] The distributed storage system can store data scattered on multiple independent devices. It can store the data as multiple copies without backing up the data, which improves the storage rate of the data and saves the storage time. Among them, the distributed storage system includes name nodes and data nodes, which are used to provide file storage services to the outside world. [0003] However, when users use a distributed storage system to store files, they need the distributed storage system to provide the "Read Your Writes" (that is, "read what you write") consistency commitment, that is, within the same process, when the user successfully writes data When the client initiates a request to read data, the written data can be read. [0004] In the prior art,...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/13G06F16/182
Inventor 周泰
Owner ALIBABA GRP HLDG LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com