478results about How to "Improve query performance" patented technology

A system and methodology for parallel query optimization using semantic-based partitioning is described. In one embodiment, for example, in a database system comprising a database storing data in database tables, a method is described for improving query performance by dynamically partitioning the data, the method comprises steps of: receiving a query requesting data from the database; generating a plurality of subplans for executing the query, each subplan including one or more operators for performing relational operations; adding operators for partitioning data and performing a given relational operation in parallel to at least some of the plurality of subplans; and building a plan for execution of the query based, at least in part, upon selecting subplans having favorable execution costs

Cached information that includes multiple network addresses is delivered in a manner that changes the order of the network addresses each time the cached information is delivered to a client, thereby achieving load balancing in a manner that reduces network traffic and improves system performance. In the preferred embodiments, the user defines a minimum time to live (TTL) and one or more order change criteria. When a query is made to a DNS, the DNS returns an answer that includes a time to live (TTL). This answer may be stored in a cache, and may include multiple network addresses. If the TTL for the answer received from a DNS is less that the minimum TTL, the TTL for the answer is set to the minimum TTL before storing the answer in the cache. When a query may be satisfied by a cached answer that includes multiple network addresses, a cache mechanism delivers the multiple cached network addresses in an order determined by the one or more order change criteria if the TTL for the cached results is a positive number. In this manner load balancing can be achieved while still benefitting from the performance enhancements of caching DNS query results.

In embodiments, methods and systems for improving a query in a database system are provided. These method and system embodiments can enable greater contextual knowledge about the types and use of data in tables underlying a relational database to be employed to improve query efficiency. By employing contextual information, embodiments can provide improved queries and / or make recommendations to a query optimizer of a database system to improve its operation based upon knowledge of the data and / or application gathered. Embodiments can be useful in improving query performance in multi-tenant database systems.

The invention discloses a method and a system for improving the large data volume query performance and belongs to the technical field of large data volume query. The method comprises A, loading data in a disk database into distributed caches in a cache ID-entity data key value pair mode, and storing the cache ID and key information of the entity data in a cache ID table of a memory database simultaneously; B, querying the cache ID table according to a query request when the query request sent by a client is obtained to selecting an ID set meeting the query request; C, obtaining the entity data from corresponding distributed caches according to the cache ID set and returning the entity data to the client. By means o the system and the method, loads of the disk database can be effectively reduced, and the big data query performance is improved.

The invention pertains to a system and method for dispatching and executing the compute-intensive parts of the workflow for database queries on an attached high-performance, parallel computing platform. The performance overhead for moving the required data and results between the database platform and the high-performance computing platform where the workload is executed is amortized in several ways, for example,by exploiting the fine-grained parallelism and superior hardware performance on the parallel computing platform for speeding up compute-intensive calculations,by using in-memory data structures on the parallel computing platform to cache data sets between a sequence of time-lagged queries on the same data, so that these queries can be processed without further data transfer overheads,by replicating data within the parallel computing platform so that multiple independent queries on the same target data set can be simultaneously processed using independent parallel partitions of the high-performance computing platform.A specific embodiment of this invention was used for deploying a bio-informatics application involving gene and protein sequence matching using the Smith-Waterman algorithm on a database system connected via an Ethernet local area network to a parallel supercomputer.

The invention provides a big data distributed storage method and system. The method comprises the steps of operating a data management engine on a main node, conducting compiling, conversion and optimization on user queries, generating and executing a query executing plan, and conducting metadata management and node monitoring; operating server processes on a distributed computational node and executing a distributed computation task; deploying the working processes of distributed computation and a single-node database on a data node; executing a subquery in the database or in a distributed computation frame. According to the big data distributed storage method and system, the opportunities that the queries are pushed down to the database to be executed are increased, data transmission cost caused by cross-node connection is avoided, and query performance is improved.

A system and method for analyzing predicate usage and proposing solutions to improve query performance. The system comprises a system for collecting performance statistics for database queries; an aggregation system that organizes the performance statistics based on predicate combinations used in the database queries;. and an evaluation system that compares the performances of queries having similar predicate combinations. Further features include a problem identification system that identifies a predicate, which when added to a query, results in a low performing query, and a solution proposal system that automatically proposes a solution to address the low performing query.

The invention relates to a method and a device for realizing a column storage based relational database. The method comprises: establishing data files, and serially numbering the data blocks which form the data files; defining a table segment; inserting records into the table segment; generating the only record identification number in the table segment for the record inserted in the table segment, and separating the records by columns; for each column in the records, executing the following operation: storing column values and the record identification numbers as value data in the data blocks and sorting according to the size of the column values; storing the record identification numbers and the serial numbers of the data blocks for storing the value data which are taken as connecting data in new data blocks, and sorting according to the size of the record identification numbers; and establishing indexes for the data blocks for storing the value data and the data blocks for storing the connecting data, and generating index data blocks. The embodiment of the invention improves the query performance of the database.

An information retrieval system implements a search language, through which a querying entity (e.g., a user, a program or process, or the like) formulates a search query. Preferably, a search query is composed of an ordered set of clause definitions, and each clause can have set membership operations applied to it. Each clause includes a clause pipeline, and a time constraint. A clause pipeline includes an ordered set of clause specifications separated by a pipeline operator. A clause specification can be either an expansion operation or a filtering operation. Preferably, a first clause specification in a pipeline operates on an initial universe of all objects, and each subsequent clause specification operates on a set of objects produced from the previous clause specification. The search language is exposed to users (typically, IT administrators), and one or more builder programs within the system (each referred to as a “model builder”) are used internally to present data models to the search language. A model builder extracts data from a given type of data source (including, without limitation, a relational database system, an application programming interface (API), or the like), and enables that data to be presented to one or more constructs of the search language according to a single unified data model.

The invention discloses a system and method for task processing. The system comprises a request receiving module, a task splitting module, a task processing module, a result integration module and a result returning module. According to the characteristic that historical data of a real-time database are managed through a non-shared mechanism in the prior art, a shared management mechanism is adopted in the system. A query task requested by a user is split into a plurality of sub-tasks by various functional modules of the system on the basis of a preset splitting strategy. Parallel processing is conducted on the sub-tasks on the basis of a data source shared mechanism. When the method is specifically implemented, a plurality of database servers can be arranged to serve as equal nodes to be connected to the real-time database, so that a data access right of the real-time database is shared. Parallel access is conducted on the real-time database, so that the sub-tasks are processed in parallel. Compared with an existing task processing method based on the non-shared management mechanism, the inquiry performance of the real-time database is greatly improved through the system and method for task processing.

The invention discloses a method and system for obtaining a dynamic feed index. Better query performance can be obtained on the occasion of high concurrent query and large data amount of an index table. An index table cache is established by a storage structure by means of a key value, and a feed index in the index table cache is consistent with a corresponding feed index in a relation type database. The method includes that a client server sends a query request to the index table cache and the relation type database, and takes query results fed back by the index table cache and the relation type database as the obtained feed index.

The invention discloses an extendible repeated data detection method, belongs to the technical field of computer storage, and solves the problem that in the existing repeated data detecting method, the storage capacity cannot be efficiently extended, so as to meet the requirements of the current situation that the storage demand increases and repeatedly deleted systems need upgrading and updating. The extendible repeated data detection method comprises the following steps: partitioning processing, fingerprint extraction, retrieving of Bloom filters, retrieving of fingerprint subset table, judgment of unfulfilled Bloom filters, new fingerprint marking, judgment of Bloom filter quantity, and extending of Bloom filter array. In the invention, the Bloom filter array is used to retrieve the fingerprint data, so as to quickly locate the retrieval range, improve the retrieval efficiency and realize detection on the repeated data; the extendible repeated data detection method is high in expansibility and querying performance, can support element location and control the misjudgment rate, and further can effectively reduce the memory overhead. The Bloom filter array is composed of a series of isomorphic Bloom filters, so that once the misjudgment rate epsilon' and the pre-established retrieving fingerprint total quantity nmax are provided, the quantity of the required Bloom filters and the number of the hush functions can be worked out.

The invention discloses a method and device for storing a relational database. The method comprises the following steps: building a control table for storing the information of a table segment, wherein the information of the table segment comprises a table segment identification as well as a minimum major key value and a maximum major key value which are stored and recorded by the table segment; when a new record is inserted into a table, inserting the new record into the memory table of the table segment of the table; when the information of the table segment is changed, correspondingly updating in the control table; when the memory occupied by all records in the memory table of the table segment reaches a predefined upper limit, storing all the records in the memory table of the table segment to a table segment data file corresponding to the table segment on a hard disk; and clearing the memory table of the table segment. According to the method and the device, the relational database can be automatically segmented and stored in lines, thereby saving the cost for storing. The storage structure of the relational database in the device is suitable to serve as the storage basis of the relational database.

The invention discloses a method and device for inquiring data. The method comprises the steps that an inquiry statement is decomposed into one or more inquiry clauses; traversal is carried out on the inquiry clauses, the inquiry clauses are matched with metadata, and the inquiry clauses are replaced with execution results stored in the metadata; otherwise, the inquiry clauses are executed, and the metadata are updated; the final execution result is used for main inquiry to obtain an inquiry result. The executed inquiry clauses and the execution results of the executed inquiry clauses are used as the metadata to be managed, the identical inquiry clauses in the current session or different sessions are directly replaced with the execution results stored in the metadata, the inquiry clauses are prevented from being executed repeatedly, and therefore the inquiry performance of the whole system is improved to a greater extent. Meanwhile, compared with optimization of the existing inquiry clause level, the inquiry optimization granularity is thinner, the method and device are more suitable for inquiring large data in data warehouse application, and therefore the inquiry performance of the whole system is further improved.

A method, apparatus and program product dynamically activate and / or deactivate buffers during execution of a database query. The dynamic deactivation and activation is based on the monitoring of information associated with the consumption of data by a plurality of threads during execution of an access plan for the database query. By doing so, the selection of a less optimal buffer position during optimization may be replaced with a more optimal buffer position during runtime, often resulting in improved query performance and parallelism.

The invention discloses a mass data querying method, belonging to the information technology field. The method comprises: a) dividing the mass data entity table according to the given time spans into sub-areas; b) establishing a two-dimensional statistical table for attribute names in the entity table, wherein one dimension represents all the given time spans in the time interval, the other dimension represents the attribute data in the attribute column of the attribute names, and the contents in the statistical table represent the name of entity table of certain attribute data existing in certain time span; c) during querying, acquiring a collection of the entity table sub-areas in the time interval according to the statistical table for the attribute name and the time interval given in the querying condition, if the querying condition comprises the statistically treated attribute name; and d) narrowing the mass data scope according to the collection and then carrying out query. The method of the invention can be used for mass data query in large-scale systems such as charging query systems of telecommunications industry, large website transaction platform and the like.

The embodiment of the invention discloses a mass data based query method and device. The mass data based query method includes: receiving an SQL which is input by a user via a system query page; performing logic analysis and physical analysis on the SQL; converting a query field required by user query into a code defined in metadata, and acquiring a query result according to a table, a storage type, and a storage physical surface in the SQL; and gathering the query results, converting the query result into a preset format, and displaying the query results in the query page. A semantic analysis technique based mass data query system can solve the problem that SQL query cannot supported while big data storage techniques such as haoop and solr are used; and a data pre-fetching technique and a Hash query technique are adopted in a system design, the query performance of the mass data query system can be optimized, and the query real-time performance of mass data can be improved.

A method to achieve acceptable query performance even when a database query optimizer chooses an inefficient query plan due to poor cardinality (row count) estimation includes generating a batch sort query plan which includes a row count operation and a reorder operation. The batch sort, and in particular the reorder operation, is not required to satisfy the query but without these operations the query plan may prove to be inefficient. When the row count operation determines that the query plan is inefficient, the row reordering operation reorders rows for more I / O efficient processing. The results of both the row count operation and the reorder operation are used to produce the desired rowset to satisfy the input query without discarding any retrieved rows. The combination of the row count operation and reorder operation allows a graceful degradation of query performance where the original query plan selected by a query optimizer without the batch sort or reorder operation would execute in an I / O inefficient manner.

The invention provides a mobile object continuous k-nearest neighbor (CKNN) query method based on a road based road networks tree (RRN-Tree) in a road network and relates to a data query method. The mobile object CKNN query method aims to solve the problem that in the prior art, an index structure is utilized to index the road network, the road network is modeled to a directed / undirected graph, and a nearest neighbor query request is processed based on a memory data structure; however, when the road network has a large data size and multiple rod segments, the query efficiency is decreased rapidly; furthermore, the modeling mode based on the graph cannot reflect the steering relation of a mobile object at the crossing, and the nearest neighbor query of the complex road network with crossing steering and U-shaped turning constraints cannot achieved. According to the mobile object CKNN query method, an RRN-Tree index structure is provided to index the road network and interesting point objects, an adjacency linked list is established for crossed points on paths in the index structure, the connection relations between road segments at the crossing are stored, and therefore, the CKNN query of the road network under complex constrain conditions is completed. The mobile object CKNN query method is used for inquiring CKNN query of the road network.

Method and system for providing results for task list queries to Workflow Management Systems (WfMSs) and Human Task Management Systems (HTMSs). In one aspect, a method for providing the result of a task list query to a human task database includes caching data from the database in a plurality of materialized views. A new request for a list of tasks from the database is received, and a new query is created when the new request applies to the data in a particular materialized view. When the particular materialized view includes data that requires updating from more recent data in the database, the materialized view is so updated. The new query is executed against the data in the materialized view instead of executing the new query against the data in the database.

The invention discloses an image retrieval system integrating classification with hash partitioning, which comprises a downloading module, a classification module training module, an image classification module, a characteristic extraction module, a recording table building module, a partitioning module, a request processing module, a retrieval module, a similarity acquiring module and a result returning module. The downloading module is used for downloading images so as to build an image library, the classification model training module classifies the images in the image library according tothe shapes and selects representative sample images from the image library to form a sample library firstly, then extracts characteristic descriptors of the classification bottom layer of all images in the sample library and performs trainings on the characteristic descriptors of the classification bottom layer by a support vector machine so as to obtain discriminant of each classification. Classification models are formed according to the discriminants of all the classifications. Precision ratio of the image retrieval system is increased, the problem of low recall ratio during classificationmistakes is overcome, and the retrieval speed of the image retrieval system is increased integrally.

The invention discloses a parallel query method for a distributed database. In the method, due to the introduction of the design of proxy and qcs modules, a query system is able to support a high-concurrency distributed database and meets requirements of a large-sized application system. Meanwhile, based on a concept of 'dividing and ruling' query tasks, the qcs module initiates query concurrently, granularity data which can be queried quickly are fed back first, and thus, the overall query performance of the system is improved obviously and the needs for quick query of a majority of application systems are met.

The invention provides a log system. The system comprises a log recording device and a log inquiring device. According to the log recording device, a recording unit records a shift quantity of a writing position of a log to a starting position of a log data file; and an index file writing unit writes a field value of every field, the log shift quantity and / or log length into corresponding index files in a correlating mode. According to the log inquiring device, a log shift quantity acquiring unit reads all log shift quantities which meet separate inquiring conditions from the corresponding index files according to a plurality of inquiring conditions; an intersection calculating unit calculates intersections of the read log shift quantities; and a log reading unit reads logs which meet the inquiring conditions from a log data file according to the log shift quantities in the intersections. By using the log system, the inquiring performance of the logs can be obviously improved, and storage space occupied by the index files can be reduced.

The invention discloses a distributed type reverse index organization method based on user log analysis. The distributed type reverse index organization method comprises the following steps: 1) analyzing query logs of the user, extracting high-frequency words and non-high-frequency words, establishing a relativity matrix of the high-frequency words, and establishing a high-frequency word relation graph according to the relativity of the high-frequency words; 2) calculating the load of each high-frequency word, and clustering the high-frequency words according to the high-frequency word relation graph and the loads of the high-frequency words; 3) distributing the clusters to nodes, establishing a high-frequency word index, hashing non-high-frequency words to the nodes, and establishing a non-high-frequency word index; 4) establishing a global index table according to the high-frequency word index and the non-high-frequency word index, and inquiring routes according to the global index table. The distributed type reverse index organization method disclosed by the invention has the advantages of small query cost, high query efficiency, and favorable query performance, and also has the advantages that the distributed type reverse index organization method can realize the balance of the throughput of the entire system and the query response speed of each time, and less nodes is referred during the query of a plurality of words.

The invention discloses a method and system for extracting a knowledge graph from software project data and answering the knowledge graph. The method comprises the following steps that for each type of software project data in a software project database, entities and an incidence relation between entities are extracted from the type of the software project data to be stored in a corresponding graph database; based on a traceability correlation technology of software data, correlation processing is carried out on the data in each graph database, and the correlation between entities of different types of software project data is obtained; and a corresponding edge is added into each graph database according to the incidence relation between entities of different types of software project data, and the entities with different sources are connected to generate the knowledge graph of the software project data; for the input natural language query statement, a matched communication sub-graphis inquired from the knowledge graph to serve as an answer. By virtue of the method and system, the problems of data correlation deletion, serious information isolation and difficulty in connection query and analysis of the software project are solved.

The invention relates to a processing method and a device of tree type data. The method includes obtaining a father node and child nodes of the tree type data; displaying the structure of the tree type data in a set mode according to the subjection relations between the father node and the child nodes and between the child nodes; and starting from one side of the boundary of the set at the outermost layer to number boundary points of each set sequentially, wherein the numbers at two sides of the boundary of each set are used for limiting the range of the set. According to the method and the device, the problem of insufficient structural management of the tree type data in the prior art is solved, the management of the tree type data is more efficient, the query performance is higher, simultaneously, limit of the tree type data hierarchy is eliminated, and the method and the device are particularly suitable for the management of mass tree type data such as a family tree, a gene spectrum and a bill of material (BOM).

The invention discloses a distributed device log collection method. According to the method, an integrated data intermediate layer is built through a distributed log processing framework in a mediator mode, integrated data intermediate management service is formed, the data intermediate management service collects device logs, the logs are stored on each distributed storage point in a distributed way, in addition, data collection is carried out, and if the distributed storage points need to be increased, the goal is achieved by adopting a distributed storage point dynamic expansion mechanism. The distributed device log collection method has the advantages that the integrated data intermediate layer is built by adopting the mediator mode, the logs are subjected to collection and formatting processing in a unified way, and the distributed storage points are uniformly managed and dispatched in a centralized way; the sub table structure is adopted, the multithreading processing advantages are better realized, a particular sub table indexing mechanism is built, and forms a super-volume data grading indexing system together with MariaDb database indexes of each data storage point, meanwhile, the performance advantage of a distributed server is utilized, and the storage and query performance of log data is greatly improved.

Previously-optimized database queries are stored in memory. When a new query needs to be optimized, the previously-optimized queries are examined to determine whether the new query has been previously optimized. If the new query has not been previously optimized, the previously-optimized queries are examined to determine whether any previously-optimized queries differ only in data type of one or more operands when compared to the new query. If a previously-optimized query that differs only in data type is located, the previously-optimized query is refreshed to reflect the different data type without the need of optimizing the new query from scratch. Portions of previously-optimized queries may thus be re-used even when a previously-optimized query is not identical to a new query to be optimized. As a result, the performance of query optimization in a database system is increased.