Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data pre-reading device based on distributed file system and method thereof

A distributed file and file reading technology, applied in transmission systems, electrical digital data processing, special data processing applications, etc., can solve the problem that the performance advantages of large-scale sequential access of data storage devices cannot be fully utilized, and the small file data cannot be effectively reduced. Read access delay and other issues to achieve the effect of reducing data access delay and access delay

Inactive Publication Date: 2014-07-09
INST OF COMPUTING TECH CHINESE ACAD OF SCI +1
View PDF1 Cites 26 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The technical problem to be solved by the present invention is to provide a data pre-reading device and method based on a distributed file system, so as to overcome the problem in the prior art that the distributed file system cannot effectively reduce the data reading of small files in the application scenario of a large number of small files. The problem of access delay and the inability to fully utilize the performance advantages of large-grained sequential access of data storage devices

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data pre-reading device based on distributed file system and method thereof
  • Data pre-reading device based on distributed file system and method thereof
  • Data pre-reading device based on distributed file system and method thereof

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0068] The specific implementation manner of the present invention is given below, and the present invention will be described in detail in conjunction with the drawings and specific implementation examples.

[0069] The invention proposes a data pre-reading method between small files. Since the data between different small files in the same directory has better spatial continuity, when reading the data of a single small file, not only the small-grained data required by the small file is obtained, but also the extension and the small file data are obtained. Spatially continuous large-grained data is pre-read from the data storage device to the cache. For data storage devices, the overhead of reading large-grained sequential data is very close to that of only reading small-grained data, so when subsequent other small files need data access, if the required data has been pre-read into the cache, it avoids data storage from the data store. The latency overhead of device synchron...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data pre-reading device based on a distributed file system. The data pre-reading device comprises a client-side module, a metadata server module and a data memory module. The client-side module obtains the catalogue read extension authorization and a small file layout by accessing the metadata server module. Small file data and large granularity data of space continuity of the small file data are pre-read to a cache of the client-side module at the same time from the data memory module according to the small file layout. The invention further discloses a data pre-reading method based on the distributed file system.

Description

technical field [0001] The invention relates to the interaction technology between a distributed file system client and a server, in particular to a method and system for a distributed file system to pre-read data between small files at the client. Background technique [0002] With the rapid development of information technology, the total amount of global data information is increasing rapidly, and there are more and more unstructured data. According to Gartner statistics, the total amount of global data information reached 1.2ZB in 2010, and it is expected to continue to grow at a high rate of at least 50% per year, 85% of which are composed of various unstructured data, which are mostly stored in the form of files. in a distributed file system. In emerging applications such as web2.0 and social networks, data information mainly exists in the form of small files, and the file size is small. With the increasing number of small files, there is an urgent need for a distrib...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): H04L29/08G06F17/30
Inventor 张军伟杨洪章邵冰清郑彩平刘振军
Owner INST OF COMPUTING TECH CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com