File metadata handler for storage and parallel processing of files in a distributed file system, and associated systems and methods

Abstract

Certain example embodiments described herein relate to a File Metadata Handler that helps an integrated managed file transfer solution perform processing in connection with complex distributed file systems, while still cooperating with more straightforward and / or conventional file systems, servers, and / or the like. The File Metadata Handler of certain example embodiments may be able to work with, or take the place of, a more conventional file handler adapter, e.g., to provide possibly required pre-processing and / or additional file processing at runtime for more complex file systems, transport protocols, and / or the like.

Description

TECHNICAL FIELD[0001]Certain example embodiments described herein relate to an integrated managed file transfer solution. More particularly, certain example embodiments described herein relate to a File Metadata Handler that helps an integrated managed file transfer solution perform processing in connection with complex distributed file systems, while still cooperating with more straightforward and / or conventional file systems, servers, and / or the like.BACKGROUND AND SUMMARY OF EXAMPLE EMBODIMENTS OF THE INVENTION[0002]Distributed File Systems are known in the computer-related arts and typically support conventional hierarchical file organizations. For instance, a user or an application can create directories and store files inside the created directories. Distributed File Systems oftentimes are set up to have master / slave architectures and oftentimes can provide high data bandwidth and scalability by using many nodes in a single cluster where every single instance can support many ...

AI Technical Summary

Benefits of technology

Distributed File Systems are used to organize and store files across multiple computers or servers. These systems use master / slave architectures and provide high data bandwidth and scalability by using many nodes in a single cluster. The systems can support millions of files. The technical effects of this innovation are improved data management and storage at a large scale.

Problems solved by technology

End users also may in some instances have to implement custom logic for processing files, thereby potentially requiring a high degree of skill, time, and in-depth knowledge about the files being processed, the systems being used, etc.

And because files typically will not be broken into pieces, individual clients oftentimes will have to process large files sequentially, thereby creating the potential for networking bottlenecks.

For example, when working with a more complex framework for distributed processing of large data sets across clusters of computers using simple programming models and / or a more distributed, scalable, and portable file system, it may not be sufficient to adapt a particular API at execution time, e.g., by providing a runtime adapter.

Although it is possible to pre-configure or hard-code constants as configuration data, etc., the application of fixed values reduces flexibility of the file system being implemented and can detract from some of the benefits provided by the more complex file system.

Images