Spark big data platform-based neighborhood density imbalance data mixed sampling method

Abstract

The invention discloses a Spark big data platform-based neighborhood density imbalance data mixed sampling method, and relates to a computer information acquisition and processing technology. According to the invention, the data is stored in the RDD through the Spark; carrying out normalization processing; The method comprises the following steps: dividing RDD into a positive domain space, a negative domain space and a boundary domain space according to neighborhood density in combination with a three-branch decision theory, sampling data of a boundary domain by adopting an SMOTE algorithm, sampling data of a negative domain by adopting a mixed sampling algorithm, and finally merging the data in the three domains to obtain a final data set. By dividing each piece of data into different domains and processing the data according to the characteristics of the different domains, a small number of types of data can be properly added, and meanwhile, most types of data are properly reduced. And finally, calling an MLLib algorithm library, and evaluating the effect by using a machine learning classifier. According to the method, the problem of inter-class proportion imbalance of unbalanceddata can be effectively alleviated, and the precision of the algorithm is improved.

Description

technical field [0001] The invention belongs to the technical fields of computer data mining and computer information processing. Background technique [0002] "Big data" has existed for a long time in the fields of physics, biology, environmental ecology, military, finance, communication and other industries, but it has attracted people's attention because of the development of the Internet and information industry in recent years. Big data is another disruptive technological revolution in the IT industry after cloud computing and the Internet of Things. Cloud computing mainly provides storage and access places and channels for data assets, and data is the real valuable asset. The amount of business transaction information within the enterprise, commodity logistics information in the Internet world, and human-to-human interaction information and location information in the Internet world will far exceed the carrying capacity of the existing enterprise IT architecture and i...

AI Technical Summary

Problems solved by technology

However, the EEG signal is very weak and is easily interfered by various environmental factors and individual factors, which has a serious impact on the performance of the EEG identification system.

The existing EEG signal preprocessing technology has many limitations, such as only suitable for processing multi-electrode EEG signals, the decomposition process is time-consuming and lacks practical guidance, and the denoising process requires manual intervention, etc.

Images