Mass data storage management method and device and data storage system

A technology for storage management and massive data, applied in the field of big data storage

Inactive Publication Date: 2017-10-20
ZTE CORP
View PDF5 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a massive data storage management method, device and data storage system to solve the existing problem of using the same storage method for all massive data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Mass data storage management method and device and data storage system
  • Mass data storage management method and device and data storage system
  • Mass data storage management method and device and data storage system

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0031] figure 1 It is a schematic structural diagram of a mass data storage management device provided in the first embodiment of the present invention, which is represented by figure 1 It can be seen that in this embodiment, the mass data storage management device 1 provided by the present invention includes:

[0032] The communication module 11 is used to receive the message to be stored, and standardize the metadata of the message to be stored;

[0033] The storage module 12 is used to store the message to be stored in the distributed file system, obtain the storage information, and store the storage information and metadata of the message to be stored in the database;

[0034] The management module 13 is configured to manage the messages stored in the distributed file system based on the storage information and metadata of the messages to be stored stored in the database.

[0035] In some embodiments, the storage module 12 in the above embodiments is configured to includ...

no. 2 example

[0041] figure 2 The flow chart of the mass data storage management method provided by the second embodiment of the present invention is represented by figure 2 It can be seen that in this embodiment, the mass data storage management method provided by the present invention includes the following steps:

[0042] S201: Receive a message to be stored, and standardize metadata of the message to be stored;

[0043] S202: Store the message to be stored in the distributed file system, obtain the storage information, and store the storage information and metadata of the message to be stored in the database;

[0044] S203: Based on the storage information and metadata of the messages to be stored stored in the database, manage the messages stored in the distributed file system.

[0045] In some embodiments, storing the message to be stored in the distributed file system in the above embodiment, and obtaining the storage information includes: storing the message to be stored and its...

no. 3 example

[0050] The present invention will be further explained in combination with specific application scenarios.

[0051] In order to solve the above problems, the present invention provides a method for separately storing massive messages according to the file size.

[0052] image 3 It shows a schematic diagram of the data storage system of the present invention, including a third-party message system, a message engine module 301, a message management module 302, a database HBASE 303, a DFS system 304 and other network elements.

[0053] based on image 3, the technical scheme that the present invention adopts is that the message engine module first preprocesses massive messages, and standardizes the metadata information of the message itself (comprising message host information, message status, message type, message source, message catalog, encoding type, message ID , message size, message subject, message delivery method, message content and other information); then according ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a mass data storage management method and device and a data storage system. The method comprises the following steps of receiving a to-be-stored message and standardizing metadata of the to-be-stored message; storing the to-be-stored message to a distributed file system to obtain stored information and storing the stored information and metadata of the to-be-stored message in a database; and managing the message stored in the distributed file system on the basis of the stored information and metadata, stored in the database, of the to-be-stored message. Through the method, after receiving the to-be-stored message, standardizing the metadata of the message and storing the metadata in the distributed file system DFS, the stored message is stored in the database, so that the message stored in the DFS system can be managed in the database; and meanwhile, the method has advantages of DFS storage and database storage at the same time, so that the problem that a same storage manner is adopted for all the mass data is solved.

Description

technical field [0001] The invention relates to the field of big data storage, in particular to a massive data storage management method, device and data storage system. Background technique [0002] "Big data" usually refers to those huge data sets that are difficult to collect, process, and analyze, and also refers to those data that are stored in traditional infrastructure for a long time. The efficient storage and management of massive messages is the cornerstone of the development of big data. Only all messages output by each business system are correctly stored, and users can display, obtain, create, save, delete, and query messages through the client / Web, and can also perform directory operations (create, rename, delete, select). [0003] With the development of science and technology, the format of messages is becoming more and more media-rich (including but not limited to audio, video, pictures, etc.), and the amount of data is increasing. There are two traditional ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/13G06F16/182G06F16/00
Inventor 刘晓晶张涛
Owner ZTE CORP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap