Index generation method and index generation device based on MapReduce programming architecture

Abstract

The invention relates to an index generation method and an index generation device based on a MapReduce programming architecture. The index generation method comprises the following steps of: acquiring data, preparing the data into a unified format and storing the prepared data in a record set formula; carrying out head encapsulation on each data record in the record set; inserting the data records subjected to data encapsulation into an HBase cluster in batch; calling a MapReduce service and an HBase service in an Hadoop cluster and connecting an Solr cluster; carrying out MapReduce operation and submitting an operation index parallel generating task to form a reverse index intermediate file; carrying out Reduce operation to generate a reverse index file; and starting a new Map task for carrying out slit operation on the reverse index file to generate a final index. According to the index generation method and the index generation device, disclosed by the invention, the storage of high-efficiency distributed mass data and the establishment of the index can be realized; and in addition, the index generation method and the index generation device have the advantages of extensibility, high fault tolerance, high performance and the like.

Description

technical field [0001] The present invention relates to the field of Internet information technology, and more specifically, to an index generation method and device based on a MapReduce programming architecture. Background technique [0002] Traditional index engines (such as: lucene and lucene-based Solr) create and manage indexes in a file-based way, which has many disadvantages: [0003] 1. It is not suitable for storage in the cluster file system, because the index files are still large batches of small files, and placing them on the cluster file system will greatly reduce the read and write efficiency; [0004] 2. It is not easy to perform cluster expansion. When a large amount of data is indexed at the same time, due to the frequent creation and merging of new fragments, it is easy to increase the response time of the index engine and reduce the efficiency. [0005] Based on the above analysis, it can be found that it is imperative to introduce an external management...

AI Technical Summary

Problems solved by technology

[0003] 1. It is not suitable for storage in the cluster file system, because the index files are still large batches of small files, and placing them on the cluster file system will greatly reduce the read and write efficiency;

[0004] 2. It is not easy to perform cluster expansion. When a large amount of data is indexed at the same time, due to the frequent creation and merging of new fragments, it is easy to increase the response time of the index engine and reduce the efficiency

Although this process does not appear to be as efficient as serial computing, the MapReduce system can handle large amounts of data that cannot be handled by general servers

Images