Mass medical data storage system and data storage method suitable for files of different sizes

Abstract

An embodiment of the invention provides a mass medical data storage system and a data storage method suitable for files of different sizes and belongs to the technical field of data storage. Aiming atapplication scenes with coexistence of mass large files and small files in the medical field, the system solves a problem that a traditional relational database is not suitable for unstructured data,a problem that redis is not suitable for mass data storage, a problem of system instability when only hbase is adopted for storage and problems of inadaptability to large files and inconvenient fileretrieval when hdfs is adopted merely for solving a problem of small file storage.

Description

Technical field [0001] The present invention relates to the technical field of data storage, and in particular to a mass medical data storage system and a data storage method suitable for large and small files. Background technique [0002] With the accelerating process of medical and health information construction, the type and scale of medical data are increasing at an unprecedented speed, and in clinical diagnosis and treatment, there is an urgent need for a large amount of knowledge to be provided by computers. At least include the following categories: One is the basic knowledge base. Mainly refers to the knowledge of "static" rules such as rational use of drugs, medical formulas, and medical term sets. The second is the clinical diagnosis and treatment knowledge base. Mainly refers to the knowledge formed through continuous manual summary, including clinical pathways, clinical guidelines, disease diagnosis and treatment knowledge base, etc. The third is reference. In p...

AI Technical Summary

Problems solved by technology

[0004] First of all, there is insufficient support for massive data and data update operations

When the amount of data is relatively large or the frequency of data reading and writing is frequent, the performance of Mysql or Orcal is very poor. Even if the lock of the database memory table is optimized, its performance will decrease as the amount of data increases;

[0005] Secondly, the types of data are more complex, and there are structured, semi-structured and unstructured data, and simply choosing a relational database becomes inapplicable

The other is to use the high-performance key-value storage system Redis database, which supports complex data types, and all data can be stored in memory, which can guarantee message Timeliness of processing, but you will encounter the following problems: Redis data generally does not require real-time landing, and it is not suitable for storing large amounts of data

Another common method to improve data storage is to choose a distributed, column-oriented Hbase database, which can build a distributed data storage cluster, but there are also the following problems: Although Hbase has better support for massive data storage, But it is not completely highly available, and when the amount of data is large, the Hbase Region jitters due to frequent splits, making storage and retrieval unstable

[0006] At present, when the amount of data is relatively large, the HDFS system is often used, but HDFS also has the problem of slowing down the entire system due to the large number of files for small files.

In order to improve this problem, at present, SequenceFile and MapFile provided by Hadoop are often used to combine small files into one large file for storage, but there are still some problems, such as the inability to simply and quickly list the directory of small files, so that fast retrieval cannot be achieved

Therefore, there are technical problems in the prior art that it is impossible to list small file directories simply and quickly, and it is impossible to realize fast retrieval

Images