Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A method and system for implementing spark cluster shared by multiple users in a python-web environment

A multi-user, cluster technology, applied in the application technology implementation field of Spark cluster, to ensure the effect of relatively fairness

Active Publication Date: 2022-04-15
COMP NETWORK INFORMATION CENT CHINESE ACADEMY OF SCI
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The present invention proposes a technical architecture aimed at explaining how to use Spark's native "inter-application scheduling" feature and Python's implementation of asynchronous, coroutine, multi-process, etc. to solve the problem of multiple Web users sharing Spark clusters in a Python development environment question

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and system for implementing spark cluster shared by multiple users in a python-web environment
  • A method and system for implementing spark cluster shared by multiple users in a python-web environment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below through specific embodiments and accompanying drawings.

[0052] The specific embodiment of the present invention is as figure 1 As shown, it includes user, UI (web site pages and corresponding functions, implemented by Django web framework), web server (Web Server) and Spark cluster (Spark Cluster). The WebSocket protocol is used for communication between the client and the Web server. figure 1 ZMQ is ZeroMQ, which is a multi-threaded network library based on message queues, which is used to realize asynchronous network communication.

[0053] The following describes the implementation involved in the present invention in detail by taking processing a task calculation request from a user as an example.

[0054] After the initial startup of the system, the asynchronous task queue (Task Queue) is empty...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a realization method and system for multi-user shared use of a Spark cluster in a Python-Web environment. The present invention introduces multiple processes to support multiple Spark contexts, and can use a limited number of Spark contexts to serve and support multi-user task computing requests; use coroutines instead of threads to guard sub-processes, and independently schedule between coroutines; Task calculation results or status are returned asynchronously to respond to user requests in real time, meeting the needs of real-time interactive use of the Web; using the characteristics of Spark inter-task scheduling, for some time-consuming tasks, the user is given the right to terminate the task , which allows web users to terminate running Spark tasks. The invention can solve the problem that multiple Web users share and use the Spark cluster in the Python development environment.

Description

technical field [0001] The invention belongs to the technical field of software design and application, and relates to an application technology implementation method, in particular to an application technology implementation method and system for multi-user interactive use of Spark clusters through the Web in a Python development environment. Background technique [0002] Apache Spark, as the most popular open source big data computing framework, is widely used in data processing and analysis applications. There are generally two ways to use it: one is to submit the user logic code packaged into a jar package through the spark-submit script; the other is to interactively process data with Spark through the spark-shell. Although these two methods are flexible enough, they are not applicable in some practical applications, such as when web users want to interact with the Spark cluster through web requests. This situation will face the problem of multiple web users sharing th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/54G06F16/25G06F16/28
CPCG06F9/546G06F16/25G06F16/284G06F2209/548
Inventor 易秀娟孟珍王学志周园春
Owner COMP NETWORK INFORMATION CENT CHINESE ACADEMY OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com