Extensible pipeline for data deduplication
A deduplication and assembly line technology, applied in the direction of electronic digital data processing, digital data information retrieval, special data processing applications, etc., can solve the problem of less value
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment Construction
[0024] Aspects of the techniques described herein generally relate to scalable pipelines for data deduplication, where individual modules / stages of the pipeline facilitate data deduplication, including by providing module chaining, module selection, main memory asynchronous processing, and / or parallel processing safe and efficient module. Typically, the various mechanisms required for deduplication (e.g., file selection, chunking, deduplication detection, compression, and commit of chunks) are each modularized in a pipeline that has a replacement for each of the individual modules, in which The ability to make selections and / or extend them.
[0025] In one aspect, the pipeline scans files using a two-stage log-based algorithm and selects files for optimization based on attributes by sorting, ranking, and / or grouping based on statistical analysis and feedback. Selected files can be processed asynchronously, in batches, and / or in parallel for data deduplication. Furthermore, t...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com