Spark task submitting method and device and server

Abstract

The invention provides a spark task submitting method, a spark task submitting device and a server. The method comprises the following steps: responding to the creation completion operation of the spark task, if the kerberos authority authentication based on the super user is passed, generating a spark-submit command containing task parameters of the spark task, and assigning an agent user name ofthe spark-sub command as a user name corresponding to a creator of a project where the spark task is located; and submitting a spark task through the spark-submit command. According to the method, kerberos permission authentication is automatically carried out while the spark task is submitted, so that the authentication efficiency is improved; the proxy user name in the spark-submit is assignedas the user name corresponding to the creator, so that the creator and the submitter of the spark task are ensured to be the same user, the user identity information is prevented from being used by other users, and the security of a big data system is ensured.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a spark task submission method, device and server. Background technique [0002] Before submitting a spark task to a big data system, kerberos authentication is usually required to ensure data security. In related technologies, the way to perform authentication is mainly that the user adds a shell script to perform authority authentication in the spark business code. In this way, the user While implementing the spark business code, it is necessary to manually increase the authority authentication logic in each spark business code, which is a heavy workload and prone to repetitive work, resulting in low authentication efficiency; moreover, this shell script-based authentication method, It is difficult to ensure the unity between the submitter of the spark task and the creator of the spark task, and it is prone to the problem of non-standard use of Kerberos permissions,...

AI Technical Summary

Problems solved by technology

[0002] Before submitting a spark task to a big data system, kerberos authentication is usually required to ensure data security. In related technologies, the way to perform authentication is mainly that the user adds a shell script to perform authority authentication in the spark business code. In this way, the user While implementing the spark business code, it is necessary to manually increase the authority authentication logic in each spark business code, which is a heavy workload and prone to repetitive work, resulting in low authentication efficiency; moreover, this shell script-based authentication method, It is difficult to ensure the unity between the submitter of the spark task and the creator of the spark task, and it is prone to the problem of non-standard use of kerberos permissions, which reduces the security of the big data system

Images