Semantic Data Storage Scheduling Methods

Abstract

The invention provides a semantic data storage scheduling method. The method comprises the following steps: a Map / Reduce parallel computing environment is established for text data analysis; the mainprocess of text mining maintains the metadata of the document and starts the working thread to wait for the text retrieval request from the sub-process of text mining; if the retrieved requested filealready exists in the cloud cache, the worker thread sends the metadata object of the file to the shunt. The invention provides a semantic data storage scheduling method, which improves the parallel framework of MAPRUDUCE from the angles of process management and buffer management, and better meets the needs of text data mining. Aiming at the irregularity of social text, semantic vectors are usedto represent and analyze the text data effectively, which is suitable for the analysis and calculation of social text mining of various scales.

Description

technical field [0001] The invention relates to big data mining, in particular to a semantic data storage scheduling method. Background technique [0002] Big data, especially social network data, contains huge commercial and social value. Effective management and utilization of these data and the value mining of data will have a huge impact on enterprises and individuals. On the other hand, while big data brings new development opportunities, it also brings many technical challenges. Traditional information processing and computing technologies have been difficult to effectively deal with the processing of big data. The effective processing of large-scale social network data faces major technical difficulties at multiple levels such as data storage and algorithm analysis. The huge amount of data makes it difficult for traditional stand-alone machine learning and data mining algorithms to complete calculations within an acceptable time, resulting in algorithm failure. Tak...

AI Technical Summary

Problems solved by technology

The huge amount of data makes it difficult for traditional stand-alone machine learning and data mining algorithms to complete calculations within an acceptable time, resulting in algorithm failure

Taking instant messaging data as an example, due to the characteristics of real-time update and variability, the dramatic increase in data volume will make the process of natural language processing and machine learning more complicated, and the overall architecture of the current parallel computing environment is not suitable for text Efficient parallel processing of data, especially process management and cache management, does not match the storage and distributed computing architecture required by text mining algorithms; moreover, the non-standard text increases the difficulty for users to understand information and discover events degree, which eventually leads to a serious imbalance in the data distribution

In addition, there is also the problem of lack of semantic representation of Chinese words, which makes traditional methods not fully applicable to the analysis and calculation of social texts, so it is difficult to realize information intelligence, human-computer interaction and automatic question answering

Images