Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and system for merging and storing massive small files

A technology of massive small files and storage systems, applied in the field of combined storage of massive small files, can solve problems such as storage structure and storage scheme defects, efficiency problems, modification, deletion defects, etc.

Active Publication Date: 2017-07-25
AEROSPACE INFORMATION
View PDF3 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Although the existing patents retrieved in the patent database have solved the problem of deployment on the Windows platform, there are serious defects in the storage structure and storage scheme. Although the files can be written and read normally, the There are serious flaws in the modification and deletion of
In addition, existing patents use a linked list structure when storing and locating header files, and there is an efficiency problem when accessing the last file

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for merging and storing massive small files
  • Method and system for merging and storing massive small files
  • Method and system for merging and storing massive small files

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0043] The reason why the continuous reading speed of batches of small files is slow is that the file headers are frequently read or the storage space of each small file on the disk is discontinuous. Therefore, in order to solve the above two defects, it is necessary to merge small files into large files and then store them. In this implementation, we refer to the merged files as library files for short.

[0044] This embodiment provides a processing flow of a method for merging and storing a large number of small files as follows: figure 1 As shown, in this embodiment, its processing steps are as follows:

[0045] Step 11, pre-establishing the map file as the positioning file index library file; wherein, the map file and the library file establish a mapping relationship according to the two-dimensional family sequence; as figure 2 As shown, a two-dimensional family sequence with the same number of rows and columns is established for each map file mapping, wherein each map f...

Embodiment 2

[0055] This embodiment provides a massive small file merge storage system, its specific implementation structure is as follows image 3 As shown, specifically, the following modules may be included: an establishment module 31, a reception module 32, a determination module 33, and a storage module 34; wherein,

[0056] Building module 31: it is used for pre-establishing the map file as the positioning file index library file; wherein, the map file and the library file establish a mapping relationship according to a two-dimensional family sequence;

[0057] Establishment module 31: it is specifically used to map each image file to establish a two-dimensional family sequence with the same number of rows and columns, wherein each image file corresponds to a library file, and each family in the two-dimensional family sequence corresponds to a library unit.

[0058] Receiving module 32: it is used for receiving small files;

[0059] Determining module 33: it is used for determinin...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a method and a system for merging and storing massive small files. The method comprises the steps of establishing a chart file as a locating file in advance to index a library file, wherein a mapping relation is established between the chart file and the library file according to a two-dimensional family sequence; receiving the small files; determining an idle storage space in the library file according to the chart file; merging and storing the small files by use of the idle storage space, wherein the chart file records positions and lengths of the small files in the library file and returns identifiers of the corresponding small files; and determining the positions and the lengths of the small files in the library file through the identifiers, which are returned by the chart file, of the small files. According to the method and the system, the problem of storage of the massive small files is solved, the file backup is easy, the occupation memory is less during running, the access speed is rapid, the occupation rate of a CPU (Central Processing Unit) is low, concurrent writing or reading can be carried out, and the method and the system bring low-cost operation and efficient processing capacity for an enterprise.

Description

technical field [0001] The invention relates to the field of data storage, in particular to a method and system for combining and storing massive small files. Background technique [0002] With the development of science and technology and the advancement of electronic equipment, we are in the era of data explosion. The number of picture data, attachments, and posts in ordinary web servers is increasing rapidly. The total amount of various files in general web servers is calculated as 1TB , the average website picture is about 30KB, and the result will be more than 35 million files. The traditional method is to store the above files directly in the server hard disk, which may easily cause some files to be damaged and cannot be read, because the huge number of files cannot be backed up, etc. Difficulties that plague enterprises and operation and maintenance personnel. [0003] Currently existing file storage solutions, such as GFS, HDFS, Luster, Ceph, mogileFS, FastDFS, Hado...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/13
Inventor 冯尔斌张学军朱兴
Owner AEROSPACE INFORMATION
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com