Method and system for achieving file uploading

A file upload and file technology, applied in the field of big data, can solve the problems of underutilization of file server bandwidth, long time consumption, inability to exert HDFS system performance, etc.

Inactive Publication Date: 2014-08-06
INSPUR BEIJING ELECTRONICS INFORMATION IND
View PDF4 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The traditional file upload method is to upload files sequentially by selecting one of the data nodes of HDFS. According to this method, the following problems exist: on the one hand, the bandwidth of the file server is not fully utilized, and on the other hand, other data nodes of HDFS are not utilized.
Therefore, using one data node to upload files often takes too long and cannot fully utilize the system performance of HDFS

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for achieving file uploading
  • Method and system for achieving file uploading

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] figure 1 The flow chart of the method for realizing file upload in the present invention, such as figure 1 shown, including:

[0042] Step 100, acquiring a predetermined number of data nodes of a distributed file system (HDFS).

[0043] In this step, obtaining the predetermined number of data nodes of HDFS specifically includes: determining the predetermined number of HDFS data nodes according to the bandwidth utilization rate of the file server, so as to obtain the predetermined number of data nodes of HDFS.

[0044] It should be noted that the bandwidth utilization rate of the file server can be obtained through the information in the historical file process of file upload, or can be obtained from the process of uploading the test file by uploading a test file, and the bandwidth utilization rate can be obtained. The method is a technical method well known to those skilled in the art, and will not be repeated here. To obtain the bandwidth utilization rate, divide th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and system for achieving file uploading. The method comprises the steps that a preset number of data nodes of a hadoop distributed file system (HDFS) are acquired, the connectivity of the acquired preset number of data nodes is detected to obtain all communicated data nodes and the number of the communicated data nodes, HDFS file uploading orders which correspond to all files and are uploaded by a file server are set, the number of the HDFS file uploading orders is counted, and the files are uploaded according to the statistic number of the HDFS filer uploading orders and the number of the communicated data nodes. According to the method, the preset number of HDFS data nodes are acquired, the HDFS file uploading orders which correspond to all the files and are uploaded by the filer server are averagely distributed to the communicated data nodes after the connectivity of the data nodes in the HDFS is detected, file uploading is achieved, the file uploading efficiency is improved, and the time consumed in the file uploading process is shortened.

Description

technical field [0001] The invention relates to the field of big data, in particular to a method and system for realizing file upload. Background technique [0002] As human society enters the information age, data has become an equally important strategic resource like water and oil. By mining massive amounts of data, the operational decisions of governments and enterprises can be based on a more scientific basis, improving decision-making efficiency, crisis response capabilities, and public service levels. Big data, or huge amount of data, refers to the effective acquisition, management, and processing of huge-scale data, making it information that helps companies make positive business decisions. [0003] Distributed File System (HDFS, referring to Hadoop Distributed File System) is designed as a distributed file system suitable for running on general-purpose hardware. HDFS is a highly fault-tolerant system, suitable for deployment on cheap machines, and provides high-t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCH04L67/06
Inventor 辛国茂亓开元赵仁明房体盈
Owner INSPUR BEIJING ELECTRONICS INFORMATION IND
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap