33 results about "Hash join" patented technology

The invention discloses a parallel query processing method for a share-nothing database cluster in a cloud computing environment. The method comprises the following steps of: firstly, segmenting a query plan to obtain sections which are executed by all data nodes; secondly, performing section processing on all data nodes to obtain Hash joins of all data nodes; and thirdly, performing Hash aggregation on control nodes by flowing of aggregated data between all data nodes to obtain a query result set. By the flowing operation of node data, the data can flow between all nodes in the query execution process, the parallel execution query process is finished, and an execution speed is greatly increased.

Minimizing both the response time to produce the first few thousand results and the overall execution time is important for interactive querying. Current join algorithms either minimize the execution time at the expense of response time or minimize response time by producing results early without optimizing the total time. Disclosed herein is a hash-based join algorithm, called early hash join, which can be dynamically configured at any point during join processing to tradeoff faster production of results for overall execution time. Varying how inputs are read has a major effect on these two factors and provide formulas that allow an optimizer to calculate the expected rate of join output and the number of I / O operations performed using different input reading strategies.

A hash integration system includes a hash join module including build and probe inputs. A hash aggregation module may aggregate on the probe input of the hash join module, and a hash table generation module may generate an integrated hash table including a record with values from the build and aggregated probe inputs. The hash join module may join the build and aggregated probe inputs to form a joined output.

The invention relates to a network training method for video data on the basis of deep learning and hash coding. The method comprises the following steps: extracting a feature matrix of a video sample by using a deep neural network; carrying out modeling by using the acquired feature matrix of the video sample as a whole body to obtain high-dimensional real value representation of the video sample; and further representing the obtained high-dimensional real value representation into binary hash coding by using a deep network.

The invention relates to a model training method based on deep learning and hash coding. The method comprises the steps of regarding partially labeled image data as training data of a network model, and representing the training data as an analog-two-value hash code through a deep network, wherein the analogy-two-value hash code refers to a simulation two-value hash code of which the value is a continuous value; regarding the obtained analog-two-value hash code as input, connecting the analog-two-value hash code to one or more task layers, and using one or more tasks to conduct training at the same time; obtaining a two-value hash code which is used for representing the training data and carries feature information capable of being retrieved based on the analog-two-value hash code.

The invention provides a Hash join method and device. The method comprises the steps of S1, acquiring first data to be operated in a first table to be operated and second data to be operated in a second table to be operated; S2, calculating the Hash value of each first datum, determining a partition number corresponding to each first datum according to the Hash value of each first datum, calculating the Hash value of each second datum, and determining a partition number corresponding to each second datum according to the Hash value of each second datum; S3, reading data with the same partition number into the same zone in a memory; S4, conducting Hash join on the first data and the second data in the same zone. By the adoption of the Hash join method and device, join efficiency can be improved.

The invention discloses a Hash join operator acceleration method and system based on an FPGA-DDR, and belongs to the field of acceleration of a memory database. The technical problem to be solved is how to realize acceleration operation in a construction stage and a detection stage of a Hash connection algorithm. The method comprises: storing a tuple, a hash table and a linked list through a DDR memory, hash calculation is conducted through an FPGA chip, the hash table and the linked list are updated, and the FPGA chip is controlled to conduct interactive communication facing the DDR memory; under the cooperation of the DDR memory and the FPGA chip, parallel multi-thread operation is executed in the construction stage and the detection stage of the hash connection to realize the construction and matching of the hash table. The system comprises a DDR memory and an FPGA chip, the DDR memory is used for storing a tuple, a hash table and a linked list, and the FPGA chip is used for storinga base address, executing hash calculation, updating the hash table and the linked list and controlling the DDR memory to perform interactive communication.

The invention relates to a face image retrieval method and device based on deep learning and Hash coding. For the problems of insufficient facial feature expression and insufficient feature discrimination in existing facial image retrieval, a facial space network and facial space loss are provided to automatically mine a facial region with discrimination in a facial image, so that the influence ofbackground information is reduced. Meanwhile, the Hash network learns the internal relation between the face features and the Hash codes, the face images are mapped into the Hash codes, and the calculation and storage cost of retrieval is remarkably reduced. The multi-scale face feature channel enhancement module in the Hash network enhances the dimension of strong distinguishing power in the face features. According to the alternate training strategy provided by the invention, the two networks are organically fused together, the information loss between face feature extraction and hash codegeneration is reduced, the distinguishing power of the face features is enhanced, meanwhile, the distinguishing power of the generated hash codes is enhanced, and the accuracy of face retrieval is improved.

The invention relates to the technical field of the method for realizing infringement resource extraction and identification comparison by using technical ways to prevent infringement copying and diffusion of picture contents, especially to a method involved in copyright protection of the picture content. The technical method is implemented as follows: step one, carrying out picture resource crawling; step two, carrying out picture feature extraction by using deep learning and coding the picture by using hash coding; and step three, carrying out picture code comparison by using a two-level hash code positioning technology to identify whether the content is an infringement content.

An embodiment of the invention relates to the technical field of databases and discloses a device and a method for data transmission among Hash join operators. The method includes: connecting an external table with a pre-generated Hash table; updating the Hash table according to the external table to acquire the updated Hash table; detecting whether a next external table exists or not; if the next external table is detected, connecting the next external table with the updated Hash table. Implementing the embodiment enables data transmission among the Hash join operators in a Hash table manner, and database query performance is improved.

The invention discloses a deep learning and Hash coding video retrieval method, and the method is characterized by comprising the following steps of: performing shot segmentation and feature extraction by using the same feature; in video retrieval, shot segmentation takes video frames of the same shot as redundant frames to be processed so as to reduce the calculated amount and improve the retrieval efficiency, and features obtained through a shot feature extraction algorithm are taken as the basis of shot segmentation so as to solve the problem that features of different frames in the same shot are different. The method has the advantages that the video retrieval at the lens level can be realized, and the retrieval accuracy can be improved; The video shot retrieval accuracy in the aspectof video retrieval capacity reaches 95% or above, pure binary operation is adopted, and the speed can be increased by four times.

The invention discloses an APUF security authentication method based on polynomial reconstruction. The method comprises two stages of registration and authentication. At the registration stage, a server randomly generates a large number of incentives and a random number k, and an APUF generates corresponding response to serve as an abscissa. After CRC encoding, a corresponding real point is foundfrom a polynomial using the k as the coefficient, and a large number of hash points are added to form a point set V. The incentives, the point set V and a polynomial coefficient hash (k) subjected tothe hash are stored in the server. At the authentication stage, a PUF circuit on the hand of a user generates response, randomizes the response, and queries the corresponding real point in the point set V for constructing a polynomial so as to obtain a polynomial coefficient (the formula is described in the specification). Contrast authentication is performed on the (the formula is described in the specification) and the hash (k), and the authentication is successful if the two coefficients are the same. According to the APUF security authentication method disclosed by the invention, by meansof the response to the randomization processing by the APUF and the adding of the hash points, the machine learning attack can be resisted to ensure the security and the reliability of the data.

The invention provides a data connection method across a data center of an HDFS (Hadoop Distributed File System). The method comprises the following steps that: a coordination node acquires the IP (Internet Protocol) addresses of metadata nodes from a configuration file after receiving a data connection operating request, establishes connection with the metadata nodes in the configuration file one by one, and acquires metadata information about a requested metadata node when a requested table is inquired in a current metadata node; the requested node filters data of the requested node according to information about requested data, screens the requested data of the requested node, and transmits the size of a result set saved by the requested node to the coordination node; the coordination node informs small result set nodes to transmit result sets saved by the small result set nodes to all large result set nodes; all the large result set nodes perform Hash join on the result sets of the large result set nodes and the result sets transmitted by all the small result set nodes to obtain result records; and the coordination node aggregates the result records. Through the method and a system, data connection across the data center can be realized.

The invention provides an electrocardiogram data display method and a terminal apparatus. The method comprises the steps of acquiring the waveform of each heart beat from electrocardiogram signals, performing dimension reduction processing on the waveforms, and mapping the waveforms in a low-dimension space into hash points corresponding to the waveforms in a one-to-one manner according to preset constraint conditions; displaying a scatter plot including the hash points. Compared with the conventional Lorenz plot which is drawn via R-R intervals and can only reflect the exception of rhythms, the scatter plot formed in the embodiments of the invention enables waveform exceptions to be discovered and enables hash points to be displayed more clearly in limited space.

A method for detecting and handling skew and spillover in in-memory hash join operations. To improve the detection of skew and spillover in parallel processing systems, a Poisson distribution of unique hash values to Units of Parallelism (UoPs) is employed to determine the number of rows per UoP and in turn, the potential of spillover at a UoP. Hash join plan options can be selected or adjusted to reduce the likelihood of spillover.

The invention discloses a data association query method and system in a RocksDB-based key-value storage mode and belongs to the technical field of operator optimization. The technical problem to be solved by the invention is how to improve the efficiency of traditional hash join calculation based on line-by-line reading and serial operation. The associated query efficiency is improved, the adoptedtechnical scheme is as follows: the method comprises the following steps: ; the method is applied to a key-value storage system. Each row of data is stored through a channel of go language, and thenthe data is read one by one to realize block reading, i.e., the data is processed by taking a block as a unit, and after the data block is received, the data is distributed to different Join Workers for parallel computing, so that the computing performance of a Hash Join operator is improved; the method specifically comprises the following steps: S1, reading data block by block; s2, all the modules are in cache communication through channels, all the modules are made to be executed asynchronously, and all the modules are made to end execution by sending end marks; and S3, calculating the mutual concurrent execution of the Join Workers.

The invention belongs to the technical field of databases, and particularly relates to a method for pushing down predicate in SQL (Structured Query Language) connection operation, a terminal and a medium. The method comprises the following steps: carrying out Hash join on a fact table and a dimension table, generating a Hash table according to the dimension table, and generating a fact table histogram according to the fact table; according to the Hash table and the fact table histogram, obtaining N pieces of predicate histogram areas, wherein N is a positive integer; transforming the N piecesof predicate histogram areas into predicates, and pushing down the predicates to a data storage layer. Compared with minimum value / maximum value and integral Hash push-down, the method can improve data filtering ability according to the fact table histogram so as to improve the calculation efficiency of the storage layer.

The invention discloses a hash hybrid acceleration system and a method for applying the same for hash joins. The system comprises a hash partition accelerator and a host processor, wherein the hash partition accelerator is integrated on a memory, is used for accelerated processing of a hash join partition period, and comprises a hash unit, a histogram unit and a shuffle unit, the hash unit is usedfor reading multiple tuples from a relation table of a partition block in the memory, and parallelly processing keys of the tuples to generate multiple hash indexes, the histogram unit is used for parallelly updating multiple copies which are stored in histogram data in the histogram unit according to the hash indexes, and integrating all the updated copies into a form with data consistency; theshuffle unit is used for determining the positions of all the tuples stored in a target address array according to the hash indexes, and copying the tuples in the relation table to a target array; thehost processor is used for processing a build period and a probe period of hash joins. Compared with an existing processor, the hash hybrid acceleration system can greatly improve the energy efficiency of hash joins.

The invention discloses a parallel Hash connection acceleration method and system based on an FPGA, and the method comprises the steps: reading a plurality of tuple data from a first data table in each set period of a Hash connection construction stage, calculating a partition number corresponding to each tuple data through a pre-division function, guiding each tuple data to a corresponding partition, and reading one piece of tuple data from each partition to construct a hash table and a chain table; reading a plurality of tuple data from the second data table in each set period of a detection stage of hash connection, calculating a partition number corresponding to each tuple data by utilizing a pre-division function, guiding each tuple data into a corresponding partition, and reading a hash table and a chain table constructed in the tuple data detection matching construction stage from each partition, wherein the construction stage and the detection stage of hash connection are both completed in the FPGA. According to the method and the device, address conflicts do not exist when the tuples among the partitions access the hash table, and the capability of processing multiple tuples in parallel is realized.

The present invention discloses the use of computers to link data records. Data records in the first plurality of data records and the second plurality of data records are hashed. Based on the hash, the first plurality of data records and the second plurality of data records are assigned to the first group and the second group, respectively. Associated pairs of packets from the first packet and the second packet are provided to threads executing on the computer processor, and different pairs are provided to different threads. Threads operate on group pairs in parallel to determine whether to join the records in the group. If the hashes associated with the data records match, the thread concatenates the two data records under consideration together. Outputs the connected data records.

A hash mixing acceleration system and a hash connection method using the same. The system includes: a hash partition accelerator, which is configured to be integrated on the memory, and is used to accelerate the processing of the hash join partition stage. The hash partition accelerator includes: a hash unit used to read multiple block relational tables from memory Tuples, and then process the keys of multiple tuples in parallel to generate multiple hash indexes; the histogram unit is used to update multiple copies of the histogram data stored in the histogram unit in parallel according to multiple hash indexes , and the updated copies are integrated into a form of data consistency; the shuffling unit is used to determine the location of each tuple stored in the target address array based on multiple hash indexes, and copy the tuples in the relational table to An array of targets; the host processor that handles the construction and probing phases of the hash join. Compared with existing processors, the disclosed hybrid acceleration system can greatly improve the energy efficiency of hash join processing.

In one aspect, a computer-implemented method includes detecting, by a server includes one or more processors, a request to perform a hash join operation on a data structure stored in a data storage device, forming a hash lookup dictionary based on lookup results in a hash table, storing the hash lookup dictionary in a cache, and probing, during a probing phase of the hash join operation, the cache.

The invention provides a method and device for training a deep neural network based on a hash coding layer. Wherein, the method includes: obtaining a pre-trained neural network model, the neural network model includes a basic feature layer; after the basic feature layer in the neural network model, inserting the first fully connected layer used for hash coding linear mapping and used as an activation function; insert a second fully-connected layer after the hash-encoded layer as an output layer; train the weights of the first fully-connected layer and the second fully-connected layer. The present invention uses the hash coding layer as the activation function of the deep neural network training, and trains the neural network through the stochastic gradient descent method to achieve the overall optimization, and directly trains the output binary coded model without relaxing the optimization conditions change.

The invention provides a data connection method across a data center of an HDFS (Hadoop Distributed File System). The method comprises the following steps that: a coordination node acquires the IP (Internet Protocol) addresses of metadata nodes from a configuration file after receiving a data connection operating request, establishes connection with the metadata nodes in the configuration file one by one, and acquires metadata information about a requested metadata node when a requested table is inquired in a current metadata node; the requested node filters data of the requested node according to information about requested data, screens the requested data of the requested node, and transmits the size of a result set saved by the requested node to the coordination node; the coordination node informs small result set nodes to transmit result sets saved by the small result set nodes to all large result set nodes; all the large result set nodes perform Hash join on the result sets of the large result set nodes and the result sets transmitted by all the small result set nodes to obtain result records; and the coordination node aggregates the result records. Through the method and a system, data connection across the data center can be realized.

The invention discloses an aircraft query method and system based on balanced hash coding. The method comprises the following steps: modeling and representing a corresponding aircraft by utilizing flight parameter data and flight condition data of each aircraft in a data set; generating a corresponding vector space for each coded aircraft to limit the balance and identifiability of hash code generation, and determining a hash code mode of the aircraft; performing hash coding on the target aircraft by using the determined hash coding mode; and carrying out matching query by utilizing hash codeinformation of the aircraft to finally obtain the aircraft matched with the target aircraft in the data set. An unsupervised learning mode without manual intervention is adopted to carry out balancedhash coding on the aircraft, so that the storage cost of the aircraft is effectively reduced, and retrieval operation is carried out based on the coded aircraft.

The invention provides a combination device containing a hash partition accelerator and a memory. The combination device comprises the memory which includes a data storage area and a logic area; the hash partition accelerator, which is integrated on the logic area of the memory and used for processing a hash join partition stage in an accelerated way, wherein the hash partition accelerator comprises a hash unit used for reading multiple tuples in a partitioned relational table from memory, and then processing keys of the multiple tuples in parallel to generate multiple hash indexes; a histogram unit used for updating a plurality of duplicates of histogram data stored in the histogram unit in parallel, and integrating each updated duplicate into a histogram data form with consistent data; and an integrating unit used for determining the position of each tuple stored in a target address array according to the multiple hash indexes, and copying the tuples in the relational table to the target array to implement the division of the relational table.

A Hash join method, device and database management system, the method comprising: when dividing a target data group during database query, using vector as a unit of quantity to divide and calculate the Hash value of the original data in a data segment, and representing the Hash value in bits; dividing the original data corresponding to the same Hash value of specified bits into the same group based on a preset grouping rule in Hash grouping, continuing to execute Hash grouping in subsequent grouping by utilizing the unspecified bits in the previous Hash grouping, and in the grouping process, according to the positions of the original data in the target data group, ranking the original data in the same group; and conducting a join operation on the grouped and ranked original data to be joined in the corresponding groups in the target data group, thus reducing the complexity of subsequent ranking of each group.

The invention belongs to the technical field of databases, and specifically relates to a method, a terminal and a medium for pushing down a predicate in an SQL connection operation, comprising the following steps: Hash joining a fact table and a dimension table, generating a Hash table according to the dimension table, The fact table generates a fact table histogram; according to the Hash table and the N predicate histogram regions obtained by the fact table histogram, the N is a positive integer; the N predicate histogram regions are converted into predicates, and push down the predicate to the data storage layer. Compared with the minimum value / maximum value and complete Hash push-down, the present invention can improve the data filtering ability according to the histogram of the fact table, thereby improving the calculation efficiency of the storage layer.