2618 results about "Distributed File System" patented technology

A dynamic file access control and management system and method in accordance with the present invention may be a proxy file management system that includes one or more file system proxy servers that provide selective access and usage management to files available from one or more file systems or sources. The present invention may embody a secure transport protocol that tunnels distributed file systems, application independent usage controls connected to files on end-user computers, dynamically merging secondary content to a requested file, and applying bandwidth management to any of the foregoing. Embodied in the various implementations of the present invention is enhanced file security. Preferably, the proxy file management system is transparent to an end-user. A dynamic content management system may also be included that selectively adds content to requested files.

The disclosed embodiments disclose techniques for executing a cloud command for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller presents a distributed-filesystem-specific capability to a client system as a file in the distributed filesystem (e.g., using a file abstraction). Upon receiving a request from the client system to access and / or operate upon this file, the client controller executes an associated cloud command. More specifically, the cloud controller initiates a specially-defined operation that accesses additional functionality for the distributed filesystem that exceeds the scope of individual reads and writes to a typical data file.

The disclosed embodiments disclose techniques for restoring an archived file in a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. Furthermore, cloud controllers may archive infrequently-accessed files in an archival cloud storage system. During operation, a cloud controller receives a request from a client system to access an archived file, and restores this archived file from the archival cloud storage system.

A method for managing a virtual distributed file system includes maintaining by a master device located in a cloud computing environment, a metadata data structure that stores metadata about locations of most updated versions of multiple file portions and security levels of the file portions, at least one file portion being stored at a user device coupled to the cloud computing environment, and maintaining by the master device, transaction of file portions to user entities, at least one user entity being hosted by the user device, based on at least metadata stored in the metadata data structure about the file portions.

The disclosed embodiments disclose techniques for restoring an archived file in a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. Furthermore, cloud controllers may archive infrequently-accessed files in an archival cloud storage system. During operation, a cloud controller receives a request from a client system to access an archived file, and restores this archived file from the archival cloud storage system.

The present invention relates to a packet analysis system and method, which enables cluster nodes to process in parallel a large quantity of packets collected in a network in an open source distribution system called Hadoop. The packet analysis system based on a Hadoop framework includes a first module for distributing and storing packet traces in a distributed file system, a second module for distributing and processing the packet traces stored in the distributed file system in a cluster of nodes executing Hadoop using a MapReduce method, and a third module for transferring the packet traces, stored in the distributed file system, to the second module so that the packet traces can be processed using the MapReduce method and outputting a result of analysis, calculated by the second module using the MapReduce method, to the distributed file system.

A NAS (Network Attaches Storage) switch authenticates a client on multiple file servers for proxy services. The NAS switch enables proxy services by successively authenticating the client on the file servers using referrals. The NAS switch further comprises a connection manager to establish connections to the client and the file servers, a referral manager to redirect the client for successive authentications, and a transaction manager to perform data transfers with the file servers on behalf of the client. The system components support DFS (Distributed File System), and communicate using a protocol dialect that supports referral mechanisms such as NFSv4 (Network File Server version 4) or CIFS (Common Internet File System). The transaction manager also performs a protocol dialect translation service when the connection manager negotiates one protocol dialect with the client, and a different protocol dialect with the file server.

A map-reduce compatible distributed file system that consists of successive component layers that each provide the basis on which the next layer is built provides transactional read-write-update semantics with file chunk replication and huge file-create rates. A primitive storage layer (storage pools) knits together raw block stores and provides a storage mechanism for containers and transaction logs. Storage pools are manipulated by individual file servers. Containers provide the fundamental basis for data replication, relocation, and transactional updates. A container location database allows containers to be found among all file servers, as well as defining precedence among replicas of containers to organize transactional updates of container contents. Volumes facilitate control of data placement, creation of snapshots and mirrors, and retention of a variety of control and policy information. Key-value stores relate keys to data for such purposes as directories, container location maps, and offset maps in compressed files.

In one embodiment, a user or client device is connected to a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories.

A computer implemented method for executing an ANSI SQL expression belonging to the SELECT-WHERE-equi-JOIN class on data residing in a distributed file system, said method comprising the steps of entering the ANSI SQL expression into a user interface; converting the ANSI SQL expression into a map-reduce program; running the map-reduce program on the distributed file system; storing the result set of the program in the distributed file system; and presenting the result set through a user interface.

The embodiment of the disclosure relates to block chain-based distributed storage, and discloses a block chain-based distributed storage method. The method includes: obtaining a file which needs to be stored, and generating a hash value and an index of the hash value for the file. The method also includes: recording the hash value in a block chain network, and storing the file in a distributed network, wherein the file is located on the basis of the hash value in the distributed network. Therefore, according to the embodiment of the disclosure, tampering proofing and the distributed storage of the file can be realized through combining a block chain and a distributed file system, thus the problem of the limited capacity of block chain nodes is solved, and the authenticity and the availability of the stored file are effectively guaranteed.

In one embodiment, a user or client device is connected to a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories.

Indirect access to local file systems is provided using storage tank protocols, allowing federation of a local file system into a distributed file system while preserving local access to the existing data in the local file system. The goal of the present system is to federate and migrate the data on a computer system with minimum disruption to applications operating on the computer system. Existing applications on a computer system continue to operate during data federation and migration and require little or no reconfiguration either when the data migration starts or when it ends. Data consistency is maintained: existing applications may modify data in the file system during migration or federation. During federation, other computer systems (or hosts) may modify the data in the file system if access control information allows them to do so. All changes in the file system are seen consistently on all hosts. Minimal downtime is required to install the present system and reconfigure the existing applications to communicate with the present system.

Limiting propagation of requests for a file in a peer to peer media sharing system may be accomplished by sharing index information associated with the file from a first network node to a second network node in the peer to peer system, updating a distance counter in the index information by the second network node to denote the sharing of the index information about the file, and setting a request propagation limit in a request packet for requesting the file by the second network node to the distance counter.

A distributed cache data system includes a cache adapter configured to reserve a designated portion of memory in at least one node in a networked cluster of machines as a contiguous space. The designated portion of memory forms a cache and the designated portion of memory includes data cells configured to store data. The cache adapter is configured to interface with the data and a distributed file system and the cache adapter is further configured to provide an interface for external clients to access the data. The cache adapter is configured to communicate to clients via a web server, and the cache adapter is further configured to direct data access requests to appropriate the data. A related process of distributing cache data is disclosed as well.

In one embodiment, a user or client device is connected to a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories.

A logical file system is described that distributes copies of files across various different physical storage resources yet provides a consistent view to the user of his or her data, regardless of which machine the user is accessing the files from, and even when the user's computer is offline. The distributed file system uses smart data redundancy to enable a virtually infinite amount of storage as long as additional storage resources are made available to the distributed file system. The result is a reliable storage system that does not necessarily tie the user's data to the user's particular computer. Instead, the user's data is associated with the user—for life—or for however long the user would like the data to be maintained, regardless of whether the user's computer or data storage components are replaced or destroyed.

The invention relates to the technical field of Internet information, in particular to a cloud storage system and an implementation method thereof. The cloud storage system comprises more than one metadata server sets and a plurality of data block storage node sets, wherein metadata server sets manage the whole distributed file system, respond an access of a client to a file, receive a writing request of a writing client and distribute the writing request to a storage block or receive a request of a reading client for reading a file and return to a position of a block corresponding to the file; and the reading client reads file blocks from corresponding blocks and combines the file blocks. In a system, the file is divided into one or more blocks; the blocks are scatteredly stored in the data block storage node sets; and storage nodes create, delete and copy the file blocks under the command of metadata servers. In the cloud storage system and the implementation method, the safety and the transmission efficiency are improved due to block redundancy distribution and block balance distribution, and the congestion probability is reduced. The cloud storage system and the implementation method are widely applied to cloud storage systems of enterprises.

In accordance with an embodiment of the present invention, a security module may be configured to provide an owner the capability to differentiate between users. In particular, the security module may be configured to generate an asymmetric read / write key pair for respectively decrypting / encrypting data for storage on a disk. The owner of the file may distribute the read key of the asymmetric key pair to a group of users that the owner has assigned read-permission for the encrypted data, i.e., a group that has read-only access. Moreover, the owner of the file may distribute the write key of the asymmetric pair to another group of users that the owner has assigned write-permission for the encrypted data, i.e., users in the write-permission group may modify the data. Alternatively, the security module may be configured to throw away the write key and not allow further re-use of the key. The security modules may also be configured to encrypt the read key for with a further key for additional protection while stored. The security module may be also configured to generate a first set of read / write key pairs for fragments of a file. Each file fragment is encrypted with a different write key from the set of read / write key pairs. The respective read keys may then be encrypted with a second long-lived key pair chosen by the owner of the file. The security module may then configured to store the encrypted file fragment and the associated encrypted read key in a storage area of a shared computer system accessible to the users of the shared computer system. The security module may also be configured to provide distribution of the required keys-either the read / write keys for direct use, or the long-lived keys for indirect use-either by means of the data owners themselves, or through use of a key distribution center.

In one embodiment, a user or client device communicates with a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories. In some embodiments, systems and methods intelligently choose whether to use copy-on-write or point-in-time copy when saving data in a snapshot version of a file whose current version is being overwritten. In some embodiments, systems and methods allow snapshot users to return from a snapshot directory to the immediate parent directory from which the user entered into the snapshot.

A serverless distributed file system manages the storage of files and directories using one or more directory groups. The directories may be managed using Byzantine-fault-tolerant groups, whereas files are managed without using Byzantine-fault-tolerant groups. Additionally, the file system may employ a hierarchical namespace to store files. Furthermore, the directory group may employ a plurality of locks to control access to objects (e.g., files and directories) in each directory.

A method of managing segments in a distributed-file system implemented by a plurality of file servers includes determining a segment of the distributed-file system controlled by a first file server for which control is to be migrated, selecting a second file server, that is different from the first file server, to take control of the segment, and moving control of the segment from the first file server to the second file server.

The consistency callback mechanisms employed by local file systems such as NTFS and distributed file systems such as DDS, NFS and CIFS are extended to provide a shared memory foundation for efficiently broadcasting real-time high definition video from a source object to large numbers of viewers via the Internet. Distributed applications such as video viewing client applications establish connections to a common distributed file system object, and then each application registers with the underlying distributed file system to receive notifications whenever the video source modifies the source object. The data required to update images maintained by viewing clients is included in notification messages. The distributed file system employs a network of proxy cache nodes. Proxy cache nodes receive notification messages (complete with image update data) and update their cached images of the source object and then retransmit the notification messages towards the viewing clients using IP multicast techniques. In this manner, the distributed file system's consistency mechanism efficiently employs network resources to enable the real-time distribution of video content streams.

The invention relates to a distributed tracking system, comprising a log collector, a message middleware cluster, a distributed stream computing cluster, a unified configuration center, a distributed file system, a database, a distributed tracking system service cluster, a distributed tracking web server system and a distributed file storage and data processing cluster. The distributed tracking system provided by the invention can efficiently analyze the operation situation of the system.

A distributed storage and parallel mining method for state monitoring data includes the steps: defining function service models of a remote substation state monitoring unit and a state monitoring communication front-end processor by means of Web service description language, and exchanging the state monitoring data of electric power equipment in an electric power wide area network environment by a simple object access protocol; storing large-scale state monitoring data redundancy in a distributed file system, creating an index table for a state monitoring data file, inserting the index table into a large-scale structural data table and querying the state monitoring data according to a query request; and generating basic data and multi-dimensional analytical data by extracting, converting and loading to built a data warehouse, and parallelly executing association rules, classification and clustered data mining algorithm by means of MapReduce task decomposition and result summary. The distributed storage and parallel mining method can be used for effectively realizing distributed data exchange, redundant storage and rapid parallel processing for state monitoring information of the mass electric power equipment in an intelligent power network environment.

The disclosed embodiments disclose techniques that facilitate of avoiding client timeouts in a distributed filesystem. Multiple cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem in a local storage pool. During operation, a cloud controller receives from a client system a request for a data block in a target file that is stored in the distributed filesystem. Although the cloud controller is already caching the requested data block, the cloud controller delays transmission of the cached data block; this additional delay gives the cloud controller more time to access uncached data blocks for the target file from a cloud storage system, thereby ensuring that subsequent requests of such data blocks do not exceed a timeout interval on the client system.

A distributed file system is disclosed. A plurality of disk servers, operating as a single distributed disk server layer, are connected to the one or more physical disks. Each disk server stores meta-data for each of the files. A plurality of lock servers, having one or more locks for each file operates as a single distributed lock server to coordinate access to the files stored by the disk server layer. A plurality of asynchronous file servers, operating as a single distributed file server layer in response to a request for a file from a user program: (i) retrieves the requested file from the disk server layer and at least one lock, associated with the retrieved file, from the lock server, and (ii) retrieves meta-data for at least one file that is related to the requested files, and at least one lock, for each file that has had its meta-data retrieved.

A storage caching method and system manages shared access to real time data files while maintaining data file coherency and consistency in a computer network including a plurality of remote computer workstations and at least one file server. The storage caching system is implemented by storage caches, which are associated with workstations, and a cache server, which is associated with a file server, where the storage caches and the cache server interface with a distributed file system to provide shared access to real time data files by remote workstations.

A virtual distributed file system for executing applications from a remote location comprises an application server for remotely storing an original copy of the applications, and a client computing device which contains an operating system for executing the applications, a communications interface configured for communicating with the application server, a virtual file system mapped to a remote file system located on the application server, and a ring buffer cache for storing the applications on the computing device. If the modules are not found locally, the modules are transferred from the application server. The computing device is from a group consisting of a thin client, personal computer, laptop or PDA. The applications are segmented into compressed modules when transferred. The ring buffer is pre-seeded with compressed modules. The computing device is usable online or offline a network. A movable profile is accessible by a user from a plurality of computing devices.

An architecture and implementation for losslessly restarting subsystems in a distributed file system is described. By partitioning functionality and logging appropriately across the kernel and user-level boundaries on a client, the user-level subsystem may be made losslessly restartable. A particular use of transactions achieves efficiency while retaining simplicity. Practical mechanisms for supporting state-based recovery in replicated state machines and like replica are described. In particular, a map assisted state transfer may include receiving one or more state updates, marshaling one or more active data-structures into a marshaled shadow, applying the received state updates to the marshaled shadow and re-instantiating the active data-structures by unmarshaling the marshaled shadow. While active data-structures may include invariance relationships, the marshaled shadow may be structured to support independence from invariance relationships between the shadows of the active data-structures, as well as efficient incremental state update application and unmarshaling to re-instantiate the active data-structures.