709 results about "Data partitioning" patented technology

A system for automating data partitioning in a parallel database includes plural nodes connected in parallel. Each node includes a database server and two databases connected thereto. Each database server includes a query optimizer. Moreover, a partitioning advisor communicates with the database server and the query optimizer. The query optimizer and the partitioning advisor include a program for recommending and evaluating data table partitions that are useful for processing a workload of query statements. The data table partitions are recommended and evaluated without requiring the data tables to be physically repartitioned.

The system and methods described herein may reduce read / write fence latencies and cache pressure related to STM metadata accesses. These techniques may leverage locality information (as reflected by the value of a respective locale guard) associated with each of a plurality of data partitions (locales) in a shared memory to elide various operations in transactional read / write fences when transactions access data in locales owned by their threads. The locale state may be disabled, free, exclusive, or shared. For a given memory access operation of an atomic transaction targeting an object in the shared memory, the system may implement the memory access operation using a contention mediation mechanism selected based on the value of the locale guard associated with the locale in which the target object resides. For example, a traditional read / write fence may be employed in some memory access operations, while other access operations may employ an optimized read / write fence.

The invention belongs to the technical field of computer storage, and provides a safe storage method based on a plurality of cloud storage systems and a system thereof. The method and the system can achieve the purpose that in the plurality of existing cloud storage systems, safety of data and metadata can be guaranteed at the same time. The safe storage method comprises a step of data writing operation and a step of data reading operation, wherein the step of data writing operation comprises the sub-steps of data partitioning, data encryption, redundancy encoding, digital signature, sharing processing, and data storage. The step of data reading operation comprises the sub-steps of metadata partition file decryption, digital signature inspection, redundancy decoding and cipher text data block decryption. The safe storage system comprises an encryption and decryption coding module, a redundancy encoding module, a secret sharing module and a digital signature module. By means of the method and the system, metadata privacy and high availability can be guaranteed, and meanwhile, a user does not need to store any metadata information for indexing and data query.

The invention discloses a data partitioning method for a distributed parallel database system. The method comprises the following steps of: establishing a fact table and a dimension table according to the constructed distributed parallel database system; inserting records of the dimension table and the fact table on different nodes according to a partitioning rule; copying the records of the dimension table to the nodes of the fact table; and deleting and updating the data. When a data set or data stream is imported or inserted into the distributed database system in a partitioning way, the relation between tables defined by a database schema can be met on each node, particularly the primary key-foreign key restrictive condition, so the data on each node has local completeness of the data. For the query processing on the connection between the tables by using the primary key-foreign key restrictive condition, the data of each node has the local completeness on the query, so dynamic repartitioning of data between the nodes is not needed; and thus the method has the advantages of preventing time-consuming network transmission of the data, shortening the query response time and improving the query efficiency.

A method and apparatus for transporting data for a data warehouse application is described. The data from an operational data store (the source database) is organized in non-overlapping data partitions. Separate execution threads read the data from the operational data store concurrently. This is followed by concurrent transformation of the data in multiple execution threads. Finally, the data is loaded into the target data warehouse concurrently using multiple execution threads. By using multiple execution threads, the data contention is reduced. Thereby the apparatus and method of the present invention achieves increased throughput.

Systems and methods are provided for knowledge discovery in spatial data as well as to systems and methods for optimizing recipes used in spatial environments such as may be found in precision agriculture. A spatial data analysis and modeling module is provided which allows users to interactively and flexibly analyze and mine spatial data. The spatial data analysis and modeling module applies spatial data mining algorithms through a number of steps. The data loading and generation module obtains or generates spatial data and allows for basic partitioning. The inspection module provides basic statistical analysis. The preprocessing module smoothes and cleans the data and allows for basic manipulation of the data. The partitioning module provides for more advanced data partitioning. The prediction module applies regression and classification algorithms on the spatial data. The integration module enhances prediction methods by combining and integrating models. The recommendation module provides the user with site-specific recommendations as to how to optimize a recipe for a spatial environment such as a fertilizer recipe for an agricultural field.

System and method for tracking statistics at the subfile level and transparently placing or migrating inactive or less active blocks of data to other storage devices. Embodiments may provide mechanisms to track statistics at the subfile level of files including, but not limited to, database files, and to transparently place or migrate inactive or less active blocks of data of the files from higher-performing, typically more expensive, storage to lower-performing, typically less expensive, storage, while placing or migrating active blocks of data of the files to higher-performing storage, based on the subfile-level statistics rather than on file-level timestamps. In some embodiments, knowledge of file structure (e.g., database file structure), for example knowledge of database partitions with header blocks and data blocks in databases using data partitioning, may be used to separate more active and less active blocks of data of files onto storage with different performance and / or other characteristics.

Version indicators within an existing range can be associated with a data partition in a distributed data store. A partition reconfiguration can be associated with one of multiple partitions in the data store, and a new version indicator that is outside the existing range can be assigned to the reconfigured partition. Additionally, a broadcast message can be sent to multiple nodes, which can include storage nodes and / or client nodes that are configured to communicate with storage nodes to access data in a distributed data store. The broadcast message can include updated location information for data in the data store. In addition, a response message can be sent to a requesting node of the multiple nodes in response to receiving from that node a message that requests updated location information for the data. The response message can include the requested updated location information.

The invention discloses a method for processing data in online business processing, which comprises the following steps: a data partitioning field of a target data table is confirmed according to a business characteristic; and the target data table is divided into a plurality of subtables according to the data partitioning field. The invention uses the method that the core data table in an online business processing system is divided into a plurality of subtables to enable the data amount of the subtables after division to be lower than that of the original core data table. When DDL operation is carried out to the subtables, the occupied space of the temporary table is small, and blocked talks are few, thus the pressure born by the system is greatly reduced, and the maintainability and the manageability of the core data table of the database are improved.

The execution environment provides for scalability where components will execute in parallel and exploit various patterns of parallelism. Dataflow applications are represented by reusable dataflow graphs called map components, while the executable version is called a prepared map. Using runtime properties the prepared map is executed in parallel with a thread allocated to each map process. The execution environment not only monitors threads, detects and corrects deadlocks, logs and controls program exceptions, but also data input and output ports of the map components are processed in parallel to take advantage of data partitioning schemes. Port implementation supports multi-state null value tokens to more accurately report exceptions. Data tokens are batched to minimize synchronization and transportation overhead and thread contention.

A system and method are disclosed that provide a simple and efficient layered video coding technique using a backward adaptive rate-distortion optimized data partitioning (RD-DP) of DCT coefficients. The video coding system may include an rate-distortion optimized data partitioning encoder and decoder. The RD-DP encoder adapts the partition point block-by-block which greatly improves the coding efficiency of the base layer bit stream without explicit transmission thereby saving the bandwidth significantly. The RD-DP decoder can also find the partition location in backward-fashion from the decoded data.

An embedded electronic device and a firmware updating method thereof are provided. The embedded electronic device includes a first storage unit and a second storage unit with a profile partition, a data partition and a system partition. In the method, a firmware package with a setup information file and a plurality of firmware code files is first downloaded into the first storage unit. The setup information file and the firmware code files are written in the profile partition and the data partition, respectively. The embedded electronic device is re-started, and a boot loader in the system partition is executed to sequentially read each firmware code file from the data partition for writing in the system partition. A write-in progress in the profile partition is updated upon completion of writing of each firmware code file.

Systems and methods that restore a failed reconfiguration of nodes in a distributed environment. By analyzing reports from read / write quorums of nodes associated with a configuration, automatic recovery for data partitions can be facilitated. Moreover, a configuration manager component tracks current configurations for replication units and determines whether a reconfiguration is required (e.g., due to node failures, node recovery, replica additions / deletions, replica moves, or replica role changes, and the like.) Reconfigurations of data activated as being replicated from an old configuration to being replicated on a new configuration may be performed in a transactionally consistent manner based on dynamic quorums associated with the new configuration and the old configuration.

Systems and methods are disclosed for partitioning data for storage in a non-volatile memory (“NVM”), such as flash memory. In some embodiments, a priority may be assigned to data being stored, and the data may be logically partitioned based on the priority. For example, a file system may identify a logical address within a first predetermined range for higher priority data and within a second predetermined range for lower priority data, such using a union file system. Using the logical address, a NVM driver can determine the priority of data being stored and can process (e.g., encode) the data based on the priority. The NVM driver can store an identifier in the NVM along with the data, and the identifier can indicate the processing techniques used on the associated data.

A system and method for separating the data needed for inter-layer prediction and data unneeded for inter-layer prediction in the bitstream. For the coded data of a picture, the decoding of the data needed for inter-layer prediction is performed independent of the data not needed for inter-layer prediction, and it is identified whether the data is needed for inter-layer prediction.

A cell-specific dictionary is applied adaptively to adequate cells, where the cell-specific dictionary subsequently optimizes the handling of frequency-partitioned multi-dimensional data. This includes improved data partitioning with super cells or adjusting resulting cells by sub-dividing very large cells and merging multiple small cells, both of which avoid the highly skewed data distribution in cells and improve the query processing. In addition, more efficient encoding is taught within a cell in case the distinct values that actually appear in that cell are much smaller than the size of the column dictionary.

The invention discloses a data security control method and a data security control platform. The method includes S1, partitioning data according to type and security classification; S2, performing encryption, identity authentication, access control, security audit, tracking and forensics on the data according to a data partitioning result, and distributing corresponding security protection tools for security protection; S3, collecting and normalizing log information generated by the security protection tools, and creating a full-life-cycle security view of the data. The data security control method and the data security control platform have the advantages that fine-grained differential full-life-cycle protection of big data can be achieved, utilization rate of the security protection tools can be increased, and the big data can be subjected to tracking and forensics.

A method and system for generating a decision-tree classifier from a training set of records, independent of the system memory size. The method includes the steps of: generating an attribute list for each attribute of the records, sorting the attribute lists for numeric attributes, and generating a decision tree by repeatedly partitioning the records using the attribute lists. For each node, split points are evaluated to determine the best split test for partitioning the records at the node. Preferably, a gini index and class histograms are used in determining the best splits. The gini index indicates how well a split point separates the records while the class histograms reflect the class distribution of the records at the node. Also, a hash table is built as the attribute list of the split attribute is divided among the child nodes, which is then used for splitting the remaining attribute lists of the node. The method reduces I / O read time by combining the read for partitioning the records at a node with the read required for determining the best split test for the child nodes. Further, it requires writes of the records only at one out of n levels of the decision tree where n> / =2. Finally, a novel data layout on disk minimizes disk seek time. The I / O optimizations work in a general environment for hierarchical data partitioning. They also work in a multi-processor environment. After the generation of the decision tree, any prior art pruning methods may be used for pruning the tree.

A method for caching data in a hybrid peer-to-peer system comprising a plurality of interconnected peer computers is disclosed. The method comprising the steps of establishing (1702) a performance criterion, arranging (1703) the hybrid peer-to-peer system as a plurality of interconnected hierarchical groups of peers by assigning each of the plurality of peers to at least one group as a first function of the performance criterion, and processing (1704) the data into a plurality of hierarchical data partitions. The method thereafter allocates (1705), as a second function of the performance criterion, each said data partition to at least one of the groups, and caches (1706) each said data partition in at least one peer in the corresponding group.

The invention discloses a method and a device for data partitioning in a key-value database. The method comprises the following steps of: resolving partitioning number information and partitioning algorithm information from a pre-stored partitioning information basic list; and running a corresponding partitioning algorithm by a partitioning callback function according to the partitioning algorithm information, wherein the corresponding partitioning algorithm determines information of partitioned sub databases of key-value data to be partitioned according to the partitioning number information and a calculation value; and the calculation value is a value relevant to the key-value data to be partitioned. The partitioning number information and the partitioning algorithm information are recorded into the partitioning information basic list; when the key-value data are subjected to operation such as storage and query, the partitioned sub databases are determined for the key-value data according to contents recorded in the partitioning information basic list; and therefore, the partitioning mode can be flexibly changed by modifying the contents in the partitioning information basic list.

The invention discloses a method for achieving dynamic partitioning and load balancing of a data-partitioning distributed environment. According to the invention, the method can achieve the local load balancing by basic data unit dynamic combination, namely dynamic partitioning, so as to make the partition granularity meet the system operation requirement. Therefore, the method solves the problemof load balancing time limitation resulting from over-large granularity of a fixed partitioning method. The method provided by the invention has wide applicability by using a partition request queue and a scarce resource queue model to analyze node loads. A local monitor provided in the invention can dynamically adjust the number of partitions after the system achieves load balance so as to achieve self-adaptive adjustment of the partitions inside nodes, thereby improving the resource utilization rate inside the nodes and further improving the work efficiency of the entire system.

The invention provides a data safety protection method based on data partitioning. According to an employed strategy, different data needing storage at a client end are reasonably partitioned into great and small data blocks, and the great and small data blocks are respectively deployed in local and non-local cloud storage systems. Compared with the prior art, the data safety protection method based on data partitioning can effectively prevent a malicious user or a cloud storage administrator from carrying out illegal stealing and tampering of privacy data of users, provides more choices for safety of different data for users and has advantages of improved system flexibility, strong practicality and easy popularization.

This invention generally relates to methods, systems, data structures and computer program code for managing spatial data, in particular very large volumes of data such as map data for a region or country. Thus we describe a method of partitioning a database of spatial data, the database including a plurality of spatial feature objections, the method comprising: reading data from said database grid cell-by-grid cell, a said grid cell comprising a cell of a grid spatially subdividing a region of spatial coverage of said database, each said grid cell including spatial feature objects; and determining a set of spatial partitions for said database, each grid cell being allocated to a said partition, responsive to a number of said spatial feature objects in each said grid cell.

A method begins by a processing module receiving data for storage and determining whether to partition the data in accordance with a data partitioning dispersed storage scheme. When the data is to be partitioned, the method continues with the processing module partitioning the data into a local data portion and a remaining data portion in accordance with the data partitioning dispersed storage scheme, dispersed storage encoding the local data portion to produce a plurality of local encoded data elements in accordance with dispersed storage encoding parameters, sending the plurality of local encoded data elements to an associated dispersed storage network (DSN) memory for storage therein, and sending the remaining data portion to another DS module.

The invention relates to a point cloud data partitioning method based on three-dimensional laser radar. The method comprises the following steps of: (1) establishing a radar coordinate system oxyz of tested vehicles; (2) preprocessing the radar data acquired by the laser radar, establishing a region of interest under the radar coordinate system oxyz and filtering out ground noise; (3) establishing an image coordinate system o'uv, and defining the mapping relation between the radar coordinate system oxyz and the image coordinate system o'uv; (4) testing vehicles on a radar cloud picture directly by using an image processing algorithm, and characterizing all tested vehicles (except the testing vehicles) in the region of interest by using a bounding box so as to obtain four vertex coordinates of the bounding box of the tested vehicles under the image coordinate system o'uv; and (5) mapping the four vertex coordinates of the bounding box under the image coordinate system o'uv in step (4) to the radar coordinate system oxyz according to the coordinate mapping model so as to obtain the shape vector parameters of the tested vehicles according to the position vector parameter of the tested vehicles. The method is high in operation efficiency, test precision and reliability, and can be widely applied to the technical field of vehicle environment sensing.

The present invention provides a flexible, dynamic database partition method and system. The method includes the steps of acquiring a data partition rule, where the data partition rule is used to identify a first relationship between a data partition condition and a database partition; establishing a second relationship between the data partition condition and a data partition key based on the data partition rule and a third relationship between the database partition and the data partition key; adding the data partition key to a data item where the data item is stored in the database based on the second relationship between the data partition condition and the data partition key; and storing the data item in the database partition based on the data partition key of the data item.

An embodiment of the invention provides a video data partitioning method and device. The method comprises steps as follows: separating audio data and video image data from video data; performing video feature detection on the audio data and the video image data to acquire one or more candidate partition points; selecting one or more target partition points from the one or more candidate partition points; partitioning the video data according to the one or more target partition points to acquire multiple pieces of video sub-data. With the adoption of the video data partitioning method and device, automatic partitioning of the video data is realized, the manual interference operation is greatly reduced, the partitioning time is greatly shortened, and the labor cost is greatly reduced.

The invention relates to a safety cloud storage method for use in a multi-cloud environment and solves the safety storage problem of cloud data. The method includes data storage and data reading; data storage includes the steps of 1, data partitioning, 2, data deduplication, 3, data encoding, 4, data encrypting, 5, trans-cloud transmission management; data reading includes the steps of 1, metadata reading, 2, reading optimizing, 3, data reading, 4, data decrypting, 5, data decoding and 6, data assembling.

The present invention pertains to differential updating of an operating system in a client device. A delta update file includes an ordered list of operations to be performed on the new install partition in memory that will port from the old install partition. A binary differential compression algorithm may be used to determine differences for the update and to compress the data being transmitted to the client device. Blocks in the partition may cyclically depend from one another. Edges are cut to break cycles. During system operation, disk images are scanned. File operations are created and a blocks vector is developed. A graph is created where the edge weights are set equal to the number of blocks. Given this, cycles are then broken and a final order resulting from a topological sort is produced.