Maximal frequent sequential pattern mining method based on distributed log

Abstract

The invention relates to a maximal frequent sequential pattern mining method based on a distributed log.The method comprises the following steps that: on the basis of a Spark distributed computation framework, extracting a local maximal frequent sequence; utilizing prefix projection to divide search space, and carrying out recursive extraction on the local maximal frequent sequence, wherein a first frequent sequence deletes infrequent items in a log sequence dataset, a database scanning scale is lowered, and meanwhile, a corresponding relationship between the frequent sequential pattern and the maximum frequent sequential pattern is used for reducing a candidate sequence number; extracting a global maximum frequent sequence; storing the local maximal frequent sequence according to different lengths, and carrying out superset detection on the sequential pattern of adjacent lengths to judge whether a superset relationship is in the presence or not; and if the superset relationship is inthe presence, deleting a redundant sequence, and extracting a global maximum frequent sequence. The maximal frequent sequential pattern mining method based on the distributed log has higher efficiency, and supports the mining of larger-scale event sequence data.

Description

technical field [0001] The invention relates to a method for mining a maximum frequent sequence pattern based on a distributed log. Background technique [0002] With the rapid development of technologies such as cloud computing, Internet of Things, and big data, distributed server systems have become the mainstream environment for various application businesses. Various types of user access and service provision make system applications more and more reliable. Analysis of user logs or system service log information is becoming more and more important. The log information in the system is distributed. The traditional method is to concentrate the log information in a distributed environment on one computer for analysis and processing, and to obtain various status information required for system operation and maintenance by mining frequent sequence patterns, but the centralized analysis and mining method will consume a huge amount of resources. communication overhead. At th...

AI Technical Summary

Problems solved by technology

[0005] Problem 1: Extracting sequential patterns from huge amounts of distributed log data, the main problem at present is that the efficiency is very low, and there is a lack of effective parallel extraction of sequential patterns

[0006] Problem 2: Existing sequential pattern mining algorithms generally need to maintain a large candidate sequence in the stage of discovering frequent itemsets. When the support threshold is low, the running time cost is high and the operating efficiency is low.

[0007] Question 3: Most of the existing research on sequential pattern mining is to study the methods of mining frequent sequence patterns and frequent closed sequence patterns, but these algorithms will mine a large number of frequent sequences when the amount of data is large

But too many frequent sequences make it difficult for users to understand user or system behavior

Images