42 results about "Concept drifting" patented technology

The invention discloses a data stream anomaly detection system based on empirical characteristics and convolution neural network. The system includes an empirical feature extraction module, which is used to identify statistical features and header features as features based on artificial experience, which play a more important role in data packet anomaly recognition; a bit stream conversion picture module used to convert the data stream into the form of two-dimensional gray-scale picture, and then through the convolutional neural network perception, the global high-level perception features are extracted; a fusion splicing module used for fusing the above modules as the data stream characteristics and identifying abnormal data streams by using the full connection layer of the neural network; a distillation model module that replaces complex networks in actual deployment; a concept drift fine-tuning module updates the detection model of concept drift; an update experience database module adding new network attacks or hidden attack instructions to the artificial experience database. The invention accurately and efficiently detects abnormal behaviors such as network failure, user misoperation, network attack and the like.

The invention relates to the technical field of intelligent information processing and discloses a method for classifying data streams in a dynamic data environment. The method comprises the following steps: partitioning the data streams; establishing different classifiers for different concept drift; storing in a characteristic data pool of the classifiers; when a new data block arrives, judging whether the concept drift occurs or not by Kullback-Leibler (KL) divergence; if the concept drift does not occur, classifying by using the classifier at the last moment; if the concept drift occurs, seeking the proper classifiers from the characteristic data pool of the classifiers by the KL divergence and classifying; and if no coincident classifier exists, training a new classifier, adding the new classifier into the characteristic data pool of the classifiers and deleting the outdated classifiers. By the method, stable and mutational concept drift can be detected simultaneously; when the concept drift occurs, classification is performed by selecting the proper classifier to guarantee the efficiency of a model; and the performance of the model is guaranteed by deleting the outdated classifiers.

The invention discloses a user score-based project recommendation method. The method comprises the following steps: in allusion to the dynamism and diversity of interests of a user in a recommendation system, effectively fusing a maintenance dose function on the basis of the sores of user projects and completing the global learning of potential interests of the user by adopting a probability topic model according to the global influences, on the interests of the user, of the time factors; and in allusion to the sensitivity, for potential scenario change, of the learning process, user personalization-oriented secondary updating learning is carried out on the interests on the basis of a concept drift problem according to the local influences, on the potential interests of the user, of the time factors; and finally calculating the degrees of supporting the projects by the user through analyzing the interests of the user, and carrying out sorting to generate a project recommendation list. According to the method, the influences, caused by the recommendation performance, of the concept drift problem can be effectively avoided and the whole recommendation quality of the system can be improved under the condition of sufficiently mining the potential interests of the user.

The invention discloses a short-text data stream classification method based on topic models and concept drift detection. The method includes: 1, acquiring an external corpus from a knowledge libraryto construct the LDA topic model; 2, dividing a short-text data stream into data blocks according to a sliding window mechanism, and using the LDA topic model to expand short text in the data blocks to obtain an expanded data stream; 3, constructing the online BTM topic model for each data block in the expanded short-text data stream, and obtaining a topic representation of each piece of short text; 4, selecting data blocks of Q topic representations to construct a classifier to use the same to predict a class label of a newly arrived data block; 5, dividing the data blocks of the Q topic representations into category clusters according to class label distribution, and calculating semantic distances between the category clusters and the newly arrived data block to judge whether concept drift occurs; and 6, updating the classifier according to a concept drift situation. The method can be used for the short-text data stream classification problem of unceasingly changed class label distribution.

The invention relates to the technical field of intelligent information processing, and discloses a visualization method for concept drift of a data stream in a dynamic data environment. The visualization method comprises the following steps: achieving static treatment on the data stream; establishing different concept presentation modes according to different concept drift modes and saving the different concept presentation modes in concept pools; and when a new data block comes, utilizing the KL divergence algorithm to search similar concept presentations in the concept pools, if the similar concept presentations exist, counting the similar concept presentations, and if not exist, adding the new data block into the concept pools to serve as a new concept to be saved. The visualization method can be used for detecting the changed drift types of various data streams, can fully analyze the concept drift process in the data stream through counting, finally, utilizes the Bays method to draw a concept drift and transition graph according to the statistic result, and visualizes the concept drift and transition graph for assisting data mining in a concept level.

The invention provides a concept drift detection method and system based on weighted sampling and electronic equipment, and the method comprises the steps: training an offline model based on historical data, carrying out the model reasoning of online data through employing the offline model, and outputting a model reasoning result; receiving the online real-time data, calculating a concept drift value based on the online real-time data and the historical data, judging whether the concept drift value is greater than a drift threshold, if so, confirming that the online real-time data has concept drift, and if not, confirming that the online real-time data does not have concept drift; when the concept drift exists in the online real-time data, updating the offline model and training data for training the offline model based on the online real-time data and the historical data; and performing model reasoning on the online data based on the updated offline model, and outputting a model reasoning result. According to the method, the drift degree of the current model can be effectively detected, the drift degree serves as a basis for model retraining / updating, and the concept drift problem of the AI model is intelligently solved.

The present disclosure provides methods and systems for coping with outliers and perfect learning in a concept-drifting environment. Data in the concept-drifting environment are time series data, and embodiments of the present disclosure uses moving windows to solve the problem of concept-drifting.

The invention discloses an adaptive network flow concept drift detection method based on information entropy, and relates to the technical field of data processing. The concept drift problem caused byflow characteristic changes due to time lapse and different network environments is solved. By using sliding window techniques, aiming at new and old data streams, discretizing the flow characteristic attributes into a plurality of branches; comparing the plurality of flow characteristics together; comparing differences of new and old data stream sliding windows by counting information entropiesof each characteristic attribute and each branch; and when specific detection is carried out, the method comprises the following steps: firstly, solving a threshold calculation formula of the size ofa sliding window by utilizing a Hoeffding boundary theory; then, determining the size of a sliding window by utilizing a Tie-buffering method; according to the method, firstly, obtaining a new data flow sliding window and an old data flow sliding window, then calculating entropy values of the new data flow sliding window and the old data flow sliding window through information entropy, finally, jusging whether concept drifting happens to data or not according to threshold values and information entropy difference values of the new window and the old window, thus concept drifting can be effectively detected, classifiers are updated, and good classification performance and generalization capacity are shown.

The invention relates to the technical field of streaming data anomaly detection and more specifically relates to an aluminum profile extrusion process stream data anomaly detection method and devicebased on an isolated forest algorithm and a storage medium. The method comprises the following steps: S10, reading original stream data of an extrusion process of an extruder, and initializing a multi-feature semi-space isolated forest model through the original stream data; S20, entering a detection period, and using the multi-feature semi-space isolation forest model to perform anomaly detectionon the stream data in the current period; S30, judging whether the detection period is finished or not; if not, returning to the step S20, updating the detection period, and if so, entering the nextstep; S40, judging whether the abnormal rate of the current period is greater than a threshold value or not, if so, indicating that concept drift exists, updating the model by using the data of the current period; if not, returning to the step S20, and entering the next period for detection until all periods are detected. The model can be updated in real time, and the problem that abnormal detection results are inaccurate due to noise and concept drift existing in streaming data is solved.

The invention discloses a multi-label data stream classification method based on incremental learning, and the method comprises the steps: step 1, an initial training stage: carrying out the modelingof a multi-label data stream into data blocks with a fixed instance number, carrying out the naive Bayes model training of each data block according to the initial data block, and obtaining a clustercenter set through employing a KMeans algorithm, wherein the trained naive Bayes classification model and the cluster center set jointly serve as a base classifier; step 2, a concept drift detection stage: when the number of the base classifiers in the naive Bayes integration model reaches a certain number in the initial learning stage, carrying out concept drift detection from the data level andthe model level respectively; step 3, an increment updating stage: when a latest data block Dt comes, updating the base classifier by using information carried by each sample in the Dt for each base classifier in the integration model, and performing instance information updating. The concept drift in the data flow can be detected in time, the situation that the algorithm performance encounters large downslide when the concept drift occurs is avoided, the latest data can be subjected to incremental learning, and the performance of the model is guaranteed.

The invention discloses a dialogue generation method fusing knowledge. The dialogue generation method comprises the following steps that S1, a knowledge graph is constructed; and S2, a dialogue modelis constructed, the dialogue model is composed of an encoder and a decoder, and the construction process comprises word encoding, knowledge encoding, double-hop entity encoding, weighted merging and decoding. According to the method, graph coding and a graph attention mechanism are introduced to carry out double-hop entity coding, and entity semantics in dialogue are better captured based on the relationship between adjacent entities; and meanwhile, in combination with concept knowledge involved in dialogue surrounded by the knowledge graph, a more reasonable reply rich in information is given, and the problem of topic concept drifting and extension in the current dialogue process is solved.

The invention provides a method and a system for classifying input data arriving one by one in time. The method comprises the steps of a) training a predetermined quantity of a group of classifiers by utilizing recent input data, of which the quantity increases from new to old in time and the real types are obtained, as learning samples; b) based on a recent classification result of the group of the classifiers, selecting the classifier with the highest classification precision for the recent input data from the group of the classifiers; and c) classifying the current input data by utilizing the selected classifier. According to the method and the system, concept drift does not need to be specially detected and can be automatically processed, and very high classification precision can be realized.

The invention provides a method and a system for acquiring concept drift amount of data distribution. The method comprises the steps of obtaining all data including training data and test data; clustering all the data by using a Gaussian mixture model to obtain a plurality of clustering clusters of all the data; respectively obtaining concept drift amounts of data distribution of all data contained in each cluster of the plurality of clusters; and obtaining the concept drift amount of the data distribution of all the data by utilizing the concept drift amount of the data distribution of all the data contained in each cluster of the plurality of clusters. Therefore, the concept drift amount of the data distribution can be accurately obtained, the change condition of the data distribution can be accurately judged, and the reliability of the system is greatly improved.

The present disclosure provides a multi-neural network classifier fusion method and device based on fuzzy complex-valued integration, A novel classifier using complex fuzzy set-valued integral as fusion algorithm is invented, That is, fuzzy complex set-valued integral classifier, which classifies data streams under the background of big data, incrementally learns the real-time classification modelof data streams with massive characteristics, and detects the conceptual drift of data streams with dynamic changes, can improve the ability of resisting conceptual drift of classification model, which is called fuzzy complex set-valued integral classifier. The invention provides a multi-neural network classifier fusion method and a device based on fuzzy complex set-valued integration, which takea fuzzy complex-valued integration algorithm as a tool, and has the following beneficial effects in particular: the algorithm converges quickly; Strong ability to resist concept drift, can control the size of the training set, wide range of applications.

An embodiment of the invention discloses a financial fraud identification method which is used for solving a problem that an existing FDS reduces a Concept drift influence and simultaneously is difficult to satisfy an instant detecting requirement. The method comprises the following steps of acquiring current transaction behavior data of a target user; and inputting the current transaction behavior data into a ground floor classifier and acquiring a determination result output by the ground floor classifier. The embodiment of the invention also provides a financial fraud identification system.

The invention discloses an online short text data stream classification method based on feature extension, which comprises the following steps of: 1, constructing a Word2vec model according to an external corpus, and obtaining a word vector set Vec; 2, vectorizing the short text data stream by using Vec, and performing text vectorization extension based on a CNN model; 3, constructing an online deep learning network for the extended text vectors; 4, introducing concept drift semaphore into neurons in the LSTM network and detecting the distribution change of the short text stream; and 5, completing the model updating of the online deep learning network and the prediction of the short text data stream. According to the method, the classification accuracy of the short text data streams can beeffectively improved, concept drift is correctly detected, the model is adjusted, and therefore the purpose of rapidly adapting to the short text data stream environment is achieved.

The invention relates to a network flow analysis method and device based on multi-source network flow data. The method comprises the following steps: deploying a pre-training classifier pool at network flow data sources, for each data source, receiving multi-source network flow data at the current moment, carrying out flow classification on the multi-source network flow data through an online classifier, carrying out feature processing and transformation on the data collected by each network flow data source every preset time interval, and carrying out feature extraction on the data collected by each network flow data source; transmitting the processed flow data features and the feature transformation matrix to a flow drift detection module; and the flow drift detection module comprises historical concept data so as to detect concept drift according to the flow data features, the feature transformation matrix and the historical concept data, and if the concept drift is detected, a multi-source deployed online classifier is reset. By adopting the method, the multi-source network flow data can be continuously and accurately analyzed in real time.

The invention discloses a multi-label text data stream classification method based on an online sequence kernel extreme learning machine. The method comprises the steps that 1, constructing a BoW model and a sliding window mechanism according to an external corpus to divide multi-label text data streams into data blocks and then vectorize the data blocks; 2, predicting the text data block Dk at the moment k by utilizing the integrated classifier model at the moment k1, and outputting a prediction result; 3, performing feature selection on the text feature set of the text data block Dk to obtain a dimension-reduced text feature set Mk; 4, judging whether concept drift or feature drift occurs or not according to the cosine similarity between the class label spaces of the text data block Dk at the moment k and the text data block Dk1 at the moment k1 and the distribution difference between the feature sets after dimension reduction; 5, constructing an online sequence kernel extreme learning machine by utilizing all texts in the text data block Dk according to the drift detection condition, and updating to an integrated classifier model at the moment k. According to the method, the classification problem of the multi-label text data flow with feature drift and concept drift is solved.

The invention discloses a concept drift detection method based on classifier diversity and a Mcdiarmid inequality. The objective of the invention is to detect whether conceptual drift occurs in a datastream by combining the inconsistency of a plurality of classifiers and the Mcdiarmid inequality. The method comprises the following steps: 1, incrementally training two individual classifiers with relatively large divergence, monitoring the diversity of the pair of classifiers for a newly coming data stream, and calculating the difference measurement of prediction results between the classifiers; 2, setting the size of the sliding window h to be n, and if the sliding window h is not full, automatically adding the difference measurement result of the latest data stream into the sliding windowh; and if the sliding window h is full, moving the initial difference measurement result out of the sliding window, and adding the latest result; 3, giving a confidence coefficient, and solving a threshold value for judging drifting through the confidence coefficient and an Mcdiarmid inequality theory. 4, associating each element in the sliding window with a weight, calculating the difference value between the weighted average value of the sliding window at the current moment and the maximum weighted average value observed at present, and comparing the difference value with the previously obtained threshold value to judge whether drifting occurs or not, so that concept drifting can be effectively detected, a classifier is updated, and better classification performance and generalization ability are shown.

The invention relates to an abnormal data flow online calibration system heurized by a brain-like hierarchical memory mechanism. The system comprises a missing data filling module (I), a dimension reduction module (II), a multi-dimensional counting Bloom filter module (III), a hierarchical memory library module (IV), an experience knowledge base module (V) and a calibrated data block module (VI).The I is used for filling up missing data in data blocks in a data stream batch processing link. The II is used for performing low-dimensional representation on the high-dimensional data in the data block. The III is used for judging whether the new data sample is abnormal . The IV is used for storing the historical data sample processed in the step II. The V is used for storing historical data samples which are processed by I but not processed by II. The III and theIV are matched with each other to be replaced, and the IV and the V are matched with each other to be updated; and the VI is usedstoring the replaced and updated new data sample. According to the online calibration system, under the condition that original data distribution is not changed, outliers, noise, missing values and the self-adaptive concept drift phenomenon are corrected in real time.

According to the data stream classification method and device, the concept drift state is accurately recognized according to the classification error rate of the integrated classifier at the current moment and the concept drift detection threshold value, the data window at the next moment is flexibly set according to the concept drift state, and if the data window is rapidly increased when concept drift does not occur, the data window can be rapidly increased. The data stream classification efficiency is improved; when it is not determined whether concept drift occurs or not, the increasing amplitude of a data window is reduced, whether concept drift occurs or not is further determined, and the influence of noise on classification accuracy is reduced; when concept drift occurs, a data window is quickly reduced, the sensitivity to sudden change type concept drift is improved, and the integrated classifier is quickly adjusted, so that the accuracy of data stream classification by the integrated classifier is integrally improved.

The invention discloses a data flow concept drift detection method based on historical model diversity, which comprises the following steps of: extracting data flow on line by adopting an on-line bagging method, constructing a basic tree in a random forest based on a diversity standard, and detecting the change of a data flow characteristic space, pre-warning possible concept drifting, identifyinga concept drifting area, identifying and processing noise by integrating drifting pre-warning and the drifting area, and finally detecting the concept drifting by an integration method of a random forest. According to the method, the concept drift detection problem is solved by using a random forest, and various new historical model storage strategies are provided, so that the problems of instance storage and how to promote future concept drift detection by using the models are solved.

The invention discloses a credit card fraud detection method based on unbalanced flow data classification. The credit card stream data classification model comprises a bagged credit card stream data oversampling mechanism, a credit card stream data multi-type concept drift processing mechanism, a credit card stream data basic classifier weighting mechanism, a credit card stream data integration pruning mechanism and a classification prediction mechanism. According to the method, credit card flow data small samples at any past moment do not need to be reserved, and credit card dynamic flow datawith unbalanced category distribution can be effectively classified and predicted so as to detect credit card fraudulent transaction data.

The invention discloses an unbalanced concept drift data stream classification method based on G-mean dynamic weighting, and belongs to the field of data stream classification. According to the method, through an online weighting mechanism, the weights of all base classifiers are updated once and are not influenced by class distribution when a new instance rather than a complete data block arrives, the creation time of the classifiers and the Gmean performance of the classifiers on the latest p data are considered when the base classifiers are updated, and therefore the classification accuracy of the base classifiers is improved. The G-mean performance is insensitive to the class distribution of the data, the importance of multiple classes and few classes can be balanced, the classification accuracy of the minority classes is improved, meanwhile, the TPR and the TNP are incrementally calculated through the time attenuation factor, and any historical data does not need to be stored; meanwhile, two elimination mechanisms are used for controlling the scale of the integrated model, and the high efficiency and accuracy of decision making are kept.

The invention provides a model adaptive training method and device, equipment, a medium and a program product, and the method comprises the steps: obtaining all operation data of an original model during actual operation, and detecting a first concept drift value of the original model during actual operation according to the operation data; and then, according to the value model and the first concept drift value, distributing each operation data to a tagged data set and / or a non-tagged data set, and when the data size of the tagged data set is greater than or equal to a preset threshold value, utilizing an adaptive training model, and according to the tagged data set and the non-tagged data set, performing adaptive training on the original model to determine a new trained model. The technical problem of how to enable the AI model to carry out adaptive training under the condition that human intervention is as little as possible is solved. The technical effects that the manual annotation amount needed when the developers update the models is reduced, and the performance stability of the models is improved in the mode of integrating the multiple models are achieved.

The invention relates to an unmarked multi-source network flow data drift detection method and device. The method comprises the following steps: deploying network flow collectors in a plurality of network flow data sources respectively, when a drift detection period is reached, obtaining network flow data collected by the network flow collectors in the drift detection period, obtaining a data feature matrix of the network flow data according to sample features of data samples, and sending the data feature matrix to a server; further obtaining a dimension reduction data feature matrix and a feature transformation matrix, and transmitting the dimension reduction data feature matrix and the feature transformation matrix to a concept drift detection module; according to the feature transformation matrix, feature transformation is carried out on the stored historical concept data to obtain a historical concept data feature matrix, and according to the JS divergence between the dimension reduction data feature matrix and the historical concept data feature matrix, a drift detection mechanism is set to detect the concept drift of the real-time unlabeled multi-source network flow data. By adopting the method, unsupervised drift detection can be carried out.

The invention discloses a credit card fraud detection method based on unbalanced flow data classification. The credit card flow data classification model of the present invention includes a bag-based credit card flow data oversampling mechanism, a credit card flow data multi-type concept drift processing mechanism, a credit card flow data basic classifier weighting mechanism, a credit card flow data integration pruning mechanism, and classification prediction mechanism. The invention does not need to keep small samples of credit card flow data at any past time, and can effectively classify and predict credit card dynamic flow data with unbalanced category distribution, so as to detect credit card fraud transaction data.

The invention discloses a data enhancement method, system and device and a computer readable storage medium. The method comprises the steps: determining whether concept drift occurs in a current data enhancement method or not; if the current data enhancement method generates concept drift, generating a target data enhancement method; judging whether the target data enhancement method meets a data enhancement performance index or not; and if the target data enhancement method meets the data enhancement performance index, performing data enhancement on the online data based on the target data enhancement method to obtain target data. According to the data enhancement method, the target data enhancement method can be generated after the current data enhancement method generates concept drift, and the target data enhancement method is applied to perform data enhancement on the online data after the target data enhancement method meets the data enhancement performance index, so that the data enhancement method for updating the online data in real time is realized; therefore, the concept drift phenomenon can be avoided, and the accuracy of data enhancement is ensured.

The invention relates to a dynamic production environment anomaly monitoring system oriented to real-time data streams. The dynamic production environment anomaly monitoring system comprises a real-time data collection module (I), a single-node sparse discrete encoder generation module (II), a single-node real-time anomaly monitoring module (III), a full-process all-node parallel real-time monitoring module (IV) and a production mode updating module (V), wherein the I is used for collecting numerical values of all nodes and integrating the numerical values into a multi-dimensional sample; theII is used for generating a plurality of sparse discrete encoders, wherein each sparse discrete encoder is used for performing sparse discrete encoding on the numerical value of each node; the III isused for judging whether the value is abnormal; the IV is used for monitoring all nodes in parallel in real time, and identifying noise interference and concept drift of each node according to the decision matrix; and the V is used for changing the production mode when it is determined that concept drift occurs. The dynamic production environment abnormity monitoring system oriented to the real-time data flow can monitor system abnormity in real time and distinguish whether noise interference or production mode change exists from the system abnormity.