Accelerating time series data base queries using dictionary based representations

Abstract

A method for accelerating time series data base queries includes segmenting an original time series of signal values into non-overlapping chunks, where a time-scale for each of the chunks is much less than the time scale of the entire time series, representing time series signal values in each chunk as a weighted superposition of atoms that are members of a shape dictionary to create a compressed time series, storing the original time series and the compressed time series into a database, determining whether a query is answerable using the compressed time series or the original time series, and whether answering the query using the compressed time series is faster. If answering the query is faster using the compressed representation, the query is executed on weight coefficients of the compressed time series to produce a query result, and the query result is translated back into an uncompressed representation.

Description

BACKGROUND[0001]1. Technical Field[0002]This disclosure is directed to methods for accessing relational data bases containing time series data.[0003]2. Discussion of Related Art[0004]Efficiently storing and querying time-series data in relational data bases is challenging. On the one hand, the relational data model does not directly support a notion of order, but does so only indirectly through timestamps. This makes operations such as interpolation very complex. On the other hand, time-series data sets, especially if derived from sensor data, can be extremely large. This is primarily due to the fact that time-series data are stored as pairs of time stamp and value. As a consequence, queries can be expensive due to high I / O costs. One way to reduce I / O cost is to store data into the database in a compressed form to reduce I / O and computational cost. However, this approach is challenging, as different data and queries might require different compression. In addition, decompression ca...

AI Technical Summary

Benefits of technology

This patent describes a method for faster and more efficient processing of time series data base queries. The method involves segmenting the original time series into smaller parts, each representing a different time scale. These parts are then compressed using a shape dictionary to save storage space. The compressed data is then stored in a database. When a query is requested, the compressed data is translated back into an uncompressed representation to perform the necessary calculations. The method also suggests using a higher compression level to obtain a faster approximate result. Overall, this method improves the speed and accuracy of processing time series data base queries.

Problems solved by technology

Efficiently storing and querying time-series data in relational data bases is challenging.

On the one hand, the relational data model does not directly support a notion of order, but does so only indirectly through timestamps.

This makes operations such as interpolation very complex.

On the other hand, time-series data sets, especially if derived from sensor data, can be extremely large.

As a consequence, queries can be expensive due to high I / O costs.

However, this approach is challenging, as different data and queries might require different compression.

In addition, decompression can significantly slow query response time.

Finally, compression must ensure that existing DBMS optimizations are fully exploited or otherwise the benefits over non-compressed storage might not be effective.

1. It should be well-suited for time and value-series, taking the implicit ordering of values into account to allow for appropriate compression.

2. Alternative representations should be possible, so that for any given query, the best representation can be chosen before executing the query

3. It should be possible to answer queries directly on the compressed representation, as otherwise decompression would become the bottleneck, potentially negating the advantage of reducing I / O through compression. In addition, needing to decompress the data implies there is sufficient memory to hold the actual non-compressed data. In most computer systems however, maim memory is the main bottleneck driving the I / O need and most of the computation time.

4. It should make full use of the security, optimization and parallelization features of the underlying data-base, as these facilities have been extensively optimized and re-inventing them would be prohibitively expensive.

While there are general methods to compress data in databases, as well as methods to compress values series outside databases, none of these methods fulfills all of the above requirements.

These techniques are not applicable to time-series, as time series, by nature, only contain a time-stamp and a single value in a row.

In this case, the relational model would be ill-suited, as it assumes a fixed scheme of columns over all records.

Therefore, storing such data in a relational model would imply a large number of missing values to force all records to the same scheme.

This method is of limited use for time-series data, as all records usually share the same number of values.

However, many of the indexing and optimization capabilities of the database can not be exploited.

In particular, all queries along the time axis are bound to be very slow.

Another drawback of these generic methods is that they usually need to decode or decompress the data before applying the query.

In practice, most time series signals do not compressible well using the Fourier transformation; for example, the metered data coming from electricity, water or transportation metering systems typically has a low frequency sampling and have shapes that are very different from sine or cosine curves.

They therefore do not compress well using a Fourier based compression.

If large portions of the data are constant, this can lead to a significant compression.

However, it is hard to apply queries directly to the compressed data, making it necessary to decompress the data first.

Another structural issue of the above mentioned compression techniques is that the compression / accuracy trade-off should be fixed once and then used throughout the application.

Usually, the higher the compression, the lesser is the reconstruction accuracy.

However, it is not obvious how to implement this technique for representing and processing time series into a relational database.

Images