477results about How to "Improve query performance" patented technology

A system and methodology for parallel query optimization using semantic-based partitioning is described. In one embodiment, for example, in a database system comprising a database storing data in database tables, a method is described for improving query performance by dynamically partitioning the data, the method comprises steps of: receiving a query requesting data from the database; generating a plurality of subplans for executing the query, each subplan including one or more operators for performing relational operations; adding operators for partitioning data and performing a given relational operation in parallel to at least some of the plurality of subplans; and building a plan for execution of the query based, at least in part, upon selecting subplans having favorable execution costs

The invention discloses a method and a system for improving the large data volume query performance and belongs to the technical field of large data volume query. The method comprises A, loading data in a disk database into distributed caches in a cache ID-entity data key value pair mode, and storing the cache ID and key information of the entity data in a cache ID table of a memory database simultaneously; B, querying the cache ID table according to a query request when the query request sent by a client is obtained to selecting an ID set meeting the query request; C, obtaining the entity data from corresponding distributed caches according to the cache ID set and returning the entity data to the client. By means o the system and the method, loads of the disk database can be effectively reduced, and the big data query performance is improved.

The invention pertains to a system and method for dispatching and executing the compute-intensive parts of the workflow for database queries on an attached high-performance, parallel computing platform. The performance overhead for moving the required data and results between the database platform and the high-performance computing platform where the workload is executed is amortized in several ways, for example,

• • by exploiting the fine-grained parallelism and superior hardware performance on the parallel computing platform for speeding up compute-intensive calculations,
  • by using in-memory data structures on the parallel computing platform to cache data sets between a sequence of time-lagged queries on the same data, so that these queries can be processed without further data transfer overheads,
  • by replicating data within the parallel computing platform so that multiple independent queries on the same target data set can be simultaneously processed using independent parallel partitions of the high-performance computing platform.

A specific embodiment of this invention was used for deploying a bio-informatics application involving gene and protein sequence matching using the Smith-Waterman algorithm on a database system connected via an Ethernet local area network to a parallel supercomputer.

A system and method for analyzing predicate usage and proposing solutions to improve query performance. The system comprises a system for collecting performance statistics for database queries; an aggregation system that organizes the performance statistics based on predicate combinations used in the database queries;. and an evaluation system that compares the performances of queries having similar predicate combinations. Further features include a problem identification system that identifies a predicate, which when added to a query, results in a low performing query, and a solution proposal system that automatically proposes a solution to address the low performing query.

An information retrieval system implements a search language, through which a querying entity (e.g., a user, a program or process, or the like) formulates a search query. Preferably, a search query is composed of an ordered set of clause definitions, and each clause can have set membership operations applied to it. Each clause includes a clause pipeline, and a time constraint. A clause pipeline includes an ordered set of clause specifications separated by a pipeline operator. A clause specification can be either an expansion operation or a filtering operation. Preferably, a first clause specification in a pipeline operates on an initial universe of all objects, and each subsequent clause specification operates on a set of objects produced from the previous clause specification. The search language is exposed to users (typically, IT administrators), and one or more builder programs within the system (each referred to as a “model builder”) are used internally to present data models to the search language. A model builder extracts data from a given type of data source (including, without limitation, a relational database system, an application programming interface (API), or the like), and enables that data to be presented to one or more constructs of the search language according to a single unified data model.

The invention discloses a system and method for task processing. The system comprises a request receiving module, a task splitting module, a task processing module, a result integration module and a result returning module. According to the characteristic that historical data of a real-time database are managed through a non-shared mechanism in the prior art, a shared management mechanism is adopted in the system. A query task requested by a user is split into a plurality of sub-tasks by various functional modules of the system on the basis of a preset splitting strategy. Parallel processing is conducted on the sub-tasks on the basis of a data source shared mechanism. When the method is specifically implemented, a plurality of database servers can be arranged to serve as equal nodes to be connected to the real-time database, so that a data access right of the real-time database is shared. Parallel access is conducted on the real-time database, so that the sub-tasks are processed in parallel. Compared with an existing task processing method based on the non-shared management mechanism, the inquiry performance of the real-time database is greatly improved through the system and method for task processing.

The invention discloses an extendible repeated data detection method, belongs to the technical field of computer storage, and solves the problem that in the existing repeated data detecting method, the storage capacity cannot be efficiently extended, so as to meet the requirements of the current situation that the storage demand increases and repeatedly deleted systems need upgrading and updating. The extendible repeated data detection method comprises the following steps: partitioning processing, fingerprint extraction, retrieving of Bloom filters, retrieving of fingerprint subset table, judgment of unfulfilled Bloom filters, new fingerprint marking, judgment of Bloom filter quantity, and extending of Bloom filter array. In the invention, the Bloom filter array is used to retrieve the fingerprint data, so as to quickly locate the retrieval range, improve the retrieval efficiency and realize detection on the repeated data; the extendible repeated data detection method is high in expansibility and querying performance, can support element location and control the misjudgment rate, and further can effectively reduce the memory overhead. The Bloom filter array is composed of a series of isomorphic Bloom filters, so that once the misjudgment rate epsilon' and the pre-established retrieving fingerprint total quantity nmax are provided, the quantity of the required Bloom filters and the number of the hush functions can be worked out.

The invention discloses a mass data querying method, belonging to the information technology field. The method comprises: a) dividing the mass data entity table according to the given time spans into sub-areas; b) establishing a two-dimensional statistical table for attribute names in the entity table, wherein one dimension represents all the given time spans in the time interval, the other dimension represents the attribute data in the attribute column of the attribute names, and the contents in the statistical table represent the name of entity table of certain attribute data existing in certain time span; c) during querying, acquiring a collection of the entity table sub-areas in the time interval according to the statistical table for the attribute name and the time interval given in the querying condition, if the querying condition comprises the statistically treated attribute name; and d) narrowing the mass data scope according to the collection and then carrying out query. The method of the invention can be used for mass data query in large-scale systems such as charging query systems of telecommunications industry, large website transaction platform and the like.

The embodiment of the invention discloses a mass data based query method and device. The mass data based query method includes: receiving an SQL which is input by a user via a system query page; performing logic analysis and physical analysis on the SQL; converting a query field required by user query into a code defined in metadata, and acquiring a query result according to a table, a storage type, and a storage physical surface in the SQL; and gathering the query results, converting the query result into a preset format, and displaying the query results in the query page. A semantic analysis technique based mass data query system can solve the problem that SQL query cannot supported while big data storage techniques such as haoop and solr are used; and a data pre-fetching technique and a Hash query technique are adopted in a system design, the query performance of the mass data query system can be optimized, and the query real-time performance of mass data can be improved.

Method and system for providing results for task list queries to Workflow Management Systems (WfMSs) and Human Task Management Systems (HTMSs). In one aspect, a method for providing the result of a task list query to a human task database includes caching data from the database in a plurality of materialized views. A new request for a list of tasks from the database is received, and a new query is created when the new request applies to the data in a particular materialized view. When the particular materialized view includes data that requires updating from more recent data in the database, the materialized view is so updated. The new query is executed against the data in the materialized view instead of executing the new query against the data in the database.

The invention discloses an image retrieval system integrating classification with hash partitioning, which comprises a downloading module, a classification module training module, an image classification module, a characteristic extraction module, a recording table building module, a partitioning module, a request processing module, a retrieval module, a similarity acquiring module and a result returning module. The downloading module is used for downloading images so as to build an image library, the classification model training module classifies the images in the image library according tothe shapes and selects representative sample images from the image library to form a sample library firstly, then extracts characteristic descriptors of the classification bottom layer of all images in the sample library and performs trainings on the characteristic descriptors of the classification bottom layer by a support vector machine so as to obtain discriminant of each classification. Classification models are formed according to the discriminants of all the classifications. Precision ratio of the image retrieval system is increased, the problem of low recall ratio during classificationmistakes is overcome, and the retrieval speed of the image retrieval system is increased integrally.

The invention relates to a processing method and a device of tree type data. The method includes obtaining a father node and child nodes of the tree type data; displaying the structure of the tree type data in a set mode according to the subjection relations between the father node and the child nodes and between the child nodes; and starting from one side of the boundary of the set at the outermost layer to number boundary points of each set sequentially, wherein the numbers at two sides of the boundary of each set are used for limiting the range of the set. According to the method and the device, the problem of insufficient structural management of the tree type data in the prior art is solved, the management of the tree type data is more efficient, the query performance is higher, simultaneously, limit of the tree type data hierarchy is eliminated, and the method and the device are particularly suitable for the management of mass tree type data such as a family tree, a gene spectrum and a bill of material (BOM).

The invention discloses a distributed device log collection method. According to the method, an integrated data intermediate layer is built through a distributed log processing framework in a mediator mode, integrated data intermediate management service is formed, the data intermediate management service collects device logs, the logs are stored on each distributed storage point in a distributed way, in addition, data collection is carried out, and if the distributed storage points need to be increased, the goal is achieved by adopting a distributed storage point dynamic expansion mechanism. The distributed device log collection method has the advantages that the integrated data intermediate layer is built by adopting the mediator mode, the logs are subjected to collection and formatting processing in a unified way, and the distributed storage points are uniformly managed and dispatched in a centralized way; the sub table structure is adopted, the multithreading processing advantages are better realized, a particular sub table indexing mechanism is built, and forms a super-volume data grading indexing system together with MariaDb database indexes of each data storage point, meanwhile, the performance advantage of a distributed server is utilized, and the storage and query performance of log data is greatly improved.