Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Text file parallel uploading method and device

A text file and independent file technology, applied in the field of big data storage, can solve the problems of slow data upload speed and inability to fully utilize the performance of the entire cluster

Active Publication Date: 2016-05-25
INSPUR BEIJING ELECTRONICS INFORMATION IND
View PDF5 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to provide a method and device for parallel uploading of text files to solve the problems in the prior art that the performance of the entire cluster cannot be fully utilized and the data upload speed is slow when writing data to HDFS

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Text file parallel uploading method and device
  • Text file parallel uploading method and device
  • Text file parallel uploading method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0031] see figure 1 , which shows a flow chart of a method for uploading text files in parallel provided by an embodiment of the present invention, which may include the following steps:

[0032] S11: Divide the text file to be uploaded into N data blocks, where N is an integer greater than 1.

[0033] Wherein, the specific value of N can be determined according to actual needs. Generally, under the premise that N is smaller than the total number of working n...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a text file parallel uploading method and device. The method includes the following steps that: a text file to be uploaded is split into N data blocks, wherein N is an integer greater than 1; and N sub threads are started, and the N sub threads are utilized to simultaneously upload the N data blocks to a distributed file system according to a one-to-one corresponding relationship. According to the method of the invention, the text file to be uploaded is split into the N data blocks, and the N sub threads are utilized to simultaneously upload the N data blocks, wherein the N sub threads are one-to-one correspondence with working nodes, and therefore, compared with a method according to the which a whole text file to be uploaded is uploaded through one working node in the prior art, the method of the invention according to which the N working nodes are utilized to simultaneously upload the N data blocks, the performance of a whole cluster can be fully utilized, and high uploading speed can be achieved.

Description

technical field [0001] The present invention relates to the technical field of big data storage, and more specifically, to a method and device for parallel uploading of text files. Background technique [0002] With the development of computer networks, the era of massive data has arrived; for the storage, analysis, management and mining of large data sets, traditional technologies (including traditional relational databases) are incapable. How to analyze and understand these data in the fastest and best way Data is imperative. Among the existing technologies and tools, the most mature and successful set of big data solutions is the Hadoop file storage computing framework and related components built on it. [0003] HDFS (Hadoop Distributed File System, Distributed File System) in the prior art, for HDFS clients, when a certain user utilizes a client to write data in HDFS, in the whole cluster, only one corresponding working node works , other working nodes are idle, and a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): H04L29/08G06F17/30
Inventor 房体盈
Owner INSPUR BEIJING ELECTRONICS INFORMATION IND
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com