File processing method, device and system based on MongoDB and server

A file processing and file technology, applied in the field of database applications, can solve problems such as the inability to store massive small files and massive metadata information, and the inability of distributed file systems to store massive small files, and achieve the effect of horizontal expansion of the system.

Inactive Publication Date: 2019-07-16
BEIJING GRIDSUM TECH CO LTD
View PDF4 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In practical applications, after a distributed file system receives a file to be stored, it usually stores its metadata information in the memory of the Name Node. However, the memory cannot store a large amount of metadata information corresponding to a large number of small files, resulting in distributed The traditional file system cannot realize the storage of massive small files

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • File processing method, device and system based on MongoDB and server
  • File processing method, device and system based on MongoDB and server
  • File processing method, device and system based on MongoDB and server

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0057] The distributed file system is a file management system based on the client / server model. The client in the distributed file system is a client device, such as a desktop computer, a notebook computer, a mobile phone, an industrial computer, etc., and a distributed file system The servers in the system usually include a storage server and a metadata server. The received data will be stored in the storage server, and the metadata information corresponding to the data will be stored in the metadata server, so that the corresponding data can be read according to the metadata information. .

[0058] HDFS (Hadoop Distribute File System), as a commonly used distributed file system, is usually used to store very large files, such as hundreds of MB, GB or even TB level files. Generally, large files will be divided into multiple blocks (Data block) for storage, each block can be 64MB by default, and each block will store multiple copies on multiple DataNodes (which are responsibl...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a file processing method, device and system based on MongoDB and a server. The method includes: presetting a plurality of data document sets and at least one metadata document set in a MongoDB; after a to-be-stored file is received, detecting that the plurality of data document sets really do not store the to-be-stored file; storing preset attribute information based on theto-be-stored file into a corresponding data document set; updating the current metadata information of the to-be-stored file to a metadata document set. The metadata information is not stored in the memory of the Name Node, the technical problem that the mass small files cannot be stored due to the fact that a Name Node memory cannot store metadata information of the mass small files is solved, meanwhile, through classified storage of the files, storage of the large files can be achieved, and system level expansion can be conveniently achieved by adding a data file set.

Description

technical field [0001] The present invention relates to the field of database application, and more specifically relates to a MongoDB-based file processing method, device, system and server. Background technique [0002] Today, with the rapid development of Internet technology, the data that users can receive during the use of computer equipment has gradually increased exponentially. Simply expanding the storage capacity of computer equipment file systems by increasing the number of hard disks has been unable to meet the current massive file data. storage requirements. [0003] In response to this situation, the distributed file system (Distributed File System, DFS) can effectively solve the problem of data storage. The distributed file system will be fixed in a certain file system in a certain place, and it will be extended to any number of places / multiple places through the Internet. A file system, the multiple nodes formed form a file system network. When users use the ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/172G06F16/14G06F16/13G06F16/907
CPCG06F16/148G06F16/172G06F16/13G06F16/907
Inventor 王晓涛
Owner BEIJING GRIDSUM TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap