Spark-based large-scale distributed DataFrame query method

Abstract

The invention discloses a Spark-based large-scale distributed DataFrame query method, which comprises the following steps: using a system framework based on a distributed computing execution engine Spark, using DataFrame as a programming model, and using Python as a programming language; in the distributed system, through encapsulating an existing query interface of a Spark native DataFrame, eliminating the incompatibility with an API of a mainstream standalone DataFrame computing library Panas; constructing a lightweight global index, and providing a plurality of distributed DataFrame query functions according to different conditions; and establishing local indexes and auxiliary indexes, so that the query performance is improved. The problems that an existing single-machine platform DataFrame is poor in expandability and cannot process large-scale data, and an existing big data processing platform distributed DataFrame query interface is not rich, poor in usability and low in performance are solved.

Description

technical field [0001] The invention relates to the technical field of distributed computing, in particular to a spark-based large-scale distributed DataFrame query method. Background technique [0002] In big data analysis applications, structured big data analysis and processing based on table models is still the most basic requirement in many industries. DataFrame is an easy-to-use table data programming model in a programming language environment. It has a good abstraction for the statistical process of data analysis, so it has received extensive attention. [0003] The traditional relational database provides a table data model oriented to SQL query, but SQL query needs to provide the support of heavyweight (heavy-weighted) database system and SQL query engine in the background, coupled with the complexity of SQL query language, so based on SQL The table data model is still not convenient enough to operate and use in the common data analysis programming language enviro...

AI Technical Summary

Problems solved by technology

[0007] Purpose of the invention: in order to solve the problem that Pandas DataFrame cannot handle large-scale data and the existing distributed DataFrame programming model of Spark lacks the flexible and easy-to-use query function, the present invention provides a kind of query method based on Spark's large-scale distributed DataFrame, the The method can efficiently query large-scale distributed DataFrames, including location-based and tag-based queries, and provides a Pandas-like DataFrame interface, which solves the problem of lack of flexible and easy-to-use query functions for distributed DataFrames under existing big data processing platforms Problems make the function of Spark DataFrame richer and more powerful

Images