Internet big data analysis and extraction method

Abstract

The invention provides an internet big data analysis and extraction method, which comprises the following steps of: 1, dividing a data object into different parts and types according to the characteristics of data to obtain a data range to be extracted; step 2, establishing a regression model, solving each parameter of the model according to the measured data, then evaluating whether the regression model can fit the measured data, and if the regression model can fit the measured data, further narrowing the range of the data to be extracted according to the independent variable; step 3, dividing the data into more than two aggregation classes according to the characteristic attributes of the data, and grouping the data to be grabbed, the elements in each aggregation class having the same characteristics; 4, calculating the similarity degree of the two pieces of data by adopting a similarity matching method; 5, the word frequency is used as a statistical index to indicate data segment information fed back by the data; and step 6, obtaining a data analysis result. The method is automatically completed by using an embedded mapping-based representation learning algorithm, and the calculation efficiency is high.

Description

technical field [0001] The invention belongs to the technical field of big data, and in particular relates to a method for analyzing and extracting Internet big data. Background technique [0002] Big data refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain period of time. It is a massive, high-growth rate that requires a new processing model to have stronger decision-making power, insight and discovery, and process optimization capabilities. and diverse information assets. [0003] At present, many web crawlers are used to grab relevant information from public websites, and then perform structured processing and storage, which may be disturbed by a large amount of useless information such as expired information and phishing website information, and the data accuracy and practicability are low. Therefore, it is necessary to study the Internet data extraction method in depth to solve the problem of im...

AI Technical Summary

Problems solved by technology

[0004] The existing big data intelligent processing system has at least the following disadvantages: the existing data technology lacks the analysis of unstructured data, a large amount of effective information is lost, and the analysis results of the business are affected; the existing data analysis and extraction rely too much on human labor. Feature extraction, low accuracy, poor calculation efficiency, slow response to user requests, affects user experience; different services usually use different data processing and feature extraction methods, resulting in a large amount of redundant data processing, and the data units of different services feature incompatible

Images