Predo-based method and system for supporting multi-data-source hybrid execution of SQL and python scripts

Abstract

The invention discloses a Predo-based method and system for supporting multi-data-source hybrid execution of SQL and python scripts. The method and the system are specifically divided into data sourcemanagement, a grammar editor, a grammar parser and a script executor. The method comprises the following steps: constructing a multi-data-source query platform based on Predo to realize dynamic addition and deletion of data sources and query data of different data sources by using the same SQL grammar; defining a set of grammar rules capable of executing SQL and python codes in a mixed manner, wherein grammatical rule is as follows: the whole script is composed of a command line code block and a python code block, the command line code block comprises an SQL statement, the command line statement takes ':' as an end mark, a python code block needs to be wrapped with '%python', that is, the python code block starts with the '%python' and ends with the '%python'. According to the method provided by the invention, the multi-data-source query process is simplified, the cross-language barrier of SQL and python codes is eliminated, data analysts do not need to process the connection problemof python and SQL queries any more, and only pay attention to the data analysis part, so that the data analysis efficiency is greatly improved.

Description

technical field [0001] The invention relates to the technical field of data analysis, in particular to a Presto-based method and system for mixed execution of SQL and python scripts supporting multiple data sources. Background technique [0002] Currently, in the field of data analysis, analysts need to perform the following steps when using python to query and analyze the data of multiple SQL-like systems: [0003] Step 1, write query SQL for different SQL-like systems; Step 2, write link execution codes for different types of SQL systems and execute the written SQL for data query; Step 3, write data conversion code to convert query results into DataFrame; Step 4 , write the data analysis code to analyze the data; Step 5, execute the python code to get the analysis result. [0004] In the whole process, only step four is the specific data analysis process, and the rest are repeated preparations for data analysis. The process is cumbersome and the operation is complicated, ...

AI Technical Summary

Problems solved by technology

[0004] In the whole process, only step 4 is the specific data analysis process, and the rest are repeated preparations for data analysis. The process is cumbersome and the operation is complicated, which greatly affects the efficiency of data analysis.

Images