Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Merging and compressing method and device for data files in key-value storage system

A data file and storage system technology, applied in the computer field, can solve the problems of large disk I/O load, waste of storage space, waste of I/O, etc., to achieve the effect of ensuring deduplication, avoiding excessive occupation, and improving utilization Effect

Active Publication Date: 2014-04-23
BEIJING QIHOO TECH CO LTD
View PDF4 Cites 24 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] (1) Has an impact on disk I / O performance
Since multiple versions of a record may exist in multiple SSTable files, in the worst case a record may exist in all SSTable files, so the SSTable file where a record is located may repeatedly participate in Compaction to eliminate the data , and the number of participations cannot be determined, which invisibly wastes I / O resources and indirectly affects read and write performance
[0011] (2) Serious waste of storage space
Therefore, every time the SSTable files of Level 0 are compacted, all the SSTable files of Level 1 must participate repeatedly, which causes excessive disk I / O load and serious I / O waste.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Merging and compressing method and device for data files in key-value storage system
  • Merging and compressing method and device for data files in key-value storage system
  • Merging and compressing method and device for data files in key-value storage system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0040] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] figure 2 A flowchart of a method for merging and compressing data files in a key-value storage system according to an embodiment of the present invention is shown. The merge compression method can be applied to a data node of a key-value storage system such as a Cassandra system, and the data file can be an SSTable file. refer to figure 2 , the combined compression method may include:

[0042] Step 202, dividing all dat...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a merging and compressing method and device for data files in a key-value storage system and belongs to the technical field of computers. The method comprises dividing all data files of data nodes into a plurality of data hierarchies to be stored, wherein the newly added data files are stored in the lowest data hierarchy; merging and compressing the data files in all data hierarchies according to a preset merging period, and moving the data files generated through merging and compressing from the current data hierarchy to the upper data hierarchy, wherein merging and compressing is to eliminating repeated data blocks in all the data files. According to the method and the device, the disk I / O resource occupancy rate can be reduced on the basis that the repeated data block elimination effect is guaranteed, and the storage space utilization rate is improved.

Description

technical field [0001] The invention relates to the technical field of computers, in particular to a method and device for merging and compressing data files in a key-value (Key-Value) storage system. Background technique [0002] In many big data storage systems, data blocks (data records) are stored in the form of Key-Value pairs. Based on the consideration of storage and read and write performance, the data addition (Append) mode is adopted in the implementation, that is, all write operations All data blocks are written into the memory table (Memtable) first, and when the Memtable reaches a certain size, the data blocks in the Memtable are sorted and written (Dump) to the ordered string table (SSTable) file on the disk. Subsequently, all or part of the SSTable files will be merged into a new SSTable file through the compaction mechanism, so as to reduce the number of data files to improve access performance and remove duplicate data to save disk space. [0003] The Cassa...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F3/06
Inventor 王锋
Owner BEIJING QIHOO TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com