Cloud computing platform and scheduling and data analysis method and system thereof
A data analysis system and cloud computing platform technology, applied in the network field, can solve the problems of low data reading and parsing efficiency, failure to achieve distributed, high real-time processing requirements, etc., to achieve low data interaction pressure, ensure load balance, Effects that improve application processing performance
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0087] The cloud computing platform and its scheduling and data analysis methods provided by the embodiments of the present invention are as follows: figure 1 As shown, as a preferred embodiment, such as Figure 4 As shown, the method for dividing and clustering the collected website datasets and / or webpage datasets provided by the embodiments of the present invention includes:
[0088] S201, using the fuzzy C-means clustering algorithm to divide the collected data set into subcategories, and defining a cluster center for each subcategory.
[0089] S202, using particle swarm optimization to find an optimal clustering center.
[0090] The method for finding the optimal clustering center by using particle swarm calculation provided by the embodiment of the present invention is as follows:
[0091] Let the category set of the data division be {C=c 1 , c 2 ,...,c l}, the corresponding set of cluster centers is {V=v 1 , v 2 ,...,v l}, then the fitness function of particle s...
Embodiment 2
[0095] The cloud computing platform and its scheduling and data analysis methods provided by the embodiments of the present invention are as follows: figure 1 As shown, as a preferred embodiment, such as Figure 5 As shown, the method for scheduling and processing website datasets and / or webpage datasets provided by the embodiments of the present invention includes:
[0096] S301. Receive a website data set and / or a web page data set to be processed, and output the stored data set according to the state of each buffer area in the first double buffer based on a read command.
[0097] S302, using the sqoop program to extract data from the database into hadoop, and using SparkSQL to read the extracted data for calculation.
[0098] S303. Perform formatting and preprocessing on the calculated data set, and output the stored formatted and preprocessed data set according to the state of each buffer area in the second double buffer based on the read command.
[0099] S304. Perform ...
Embodiment 3
[0102] The cloud computing platform and its scheduling and data analysis methods provided by the embodiments of the present invention are as follows: figure 1 As shown, as a preferred embodiment, the data set security detection method provided by the embodiment of the present invention includes:
[0103] (1) Receive and acquire the clustered website data set and / or web page data set by using the security detection program through the security detection module; scan and identify the sensitive data in the acquired data set, analyze the data set, and extract the The source address and business type identification of the data set;
[0104] (2) Obtaining a TCP connection record corresponding to the service type identifier of the data set;
[0105] (3) extracting the TCP connection state corresponding to the source address according to the obtained TCP connection record;
[0106] (4) Judging whether the TCP connection state is normal, if so, then judging that the data set is a saf...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com