Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Metadata updating method and device based on column storage of distributed file system as well as host

A technology for distributed files and file metadata, applied in the field of data processing, can solve the problems of not being able to provide a method for dynamic modification of metadata, consuming huge computing resource overhead and time overhead, etc.

Active Publication Date: 2015-08-26
HUAWEI TECH CO LTD
View PDF4 Cites 20 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the purpose of the present invention is to solve the problem that in the existing Append-Only file system, an effective method for dynamically modifying metadata cannot be provided, and the update operation of metadata needs to consume huge computing resource overhead and time overhead. The scheme is as follows:

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Metadata updating method and device based on column storage of distributed file system as well as host
  • Metadata updating method and device based on column storage of distributed file system as well as host
  • Metadata updating method and device based on column storage of distributed file system as well as host

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0105] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0106] see figure 1 , an embodiment of the present invention provides a method for updating metadata based on columnar storage of a distributed file system, the method includes:

[0107] Step 110: Obtain the metadata to be updated in the data table of the distributed file system, divide the data records of the data table into multiple row groups by row, convert the data table into a global file metadata, and multiple row group files .

[0108] Wherein, the r...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

A method, device and host for updating metadata stored in columns in a distributed file system, the method comprising: obtaining the metadata to be updated in a data table in the distributed file system; segmenting the data records in the data table by row into a plurality of row groups; converting the data table to one global file metadata and a plurality of row group files (110), the row group files comprising actual data blocks, data index blocks, local metadata blocks, metadata index blocks and file footers; determining whether the metadata to be updated belongs to the global file metadata (120); if not, then updating the local metadata (130); adding the updated local metadata blocks, the metadata index blocks and the file footers to the a plurality of row group files according to the updated local metadata(140). The method realizes dynamic updating of the metadata, greatly saves the execution time of such updating operation and the required calculation resources.

Description

technical field [0001] The present invention relates to the field of data processing, and more specifically relates to a method, device, and host for updating metadata based on columnar storage of a distributed file system. Background technique [0002] Append-Only file system, that is, the file system only supports file read or append write operations, unlike traditional file systems that support in-place file modification operations. This type of file system has the characteristics of high write performance and easy consistency of multiple copies, and is widely used in large-scale distributed storage systems. Typical representative examples include GFS (Google File System, Google Distributed File System), HDFS (Hadoop Distributed File System, Hadoop Distributed File System), etc. In the Append-Only file system, columnar storage has unique advantages over traditional row-based storage. In columnar storage, data records are divided into columns and stored independently, so ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F3/0638G06F3/061G06F3/067G06F16/164G06F16/13G06F16/182G06F16/196G06F16/1805
Inventor 邓利群朱俊华
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com