A real-time indexing method based on inverted index

Abstract

The invention provides a real-time indexing method based on a reverse index. The method comprises the following steps: pretreating data; updating an internal memory index database; updating a disk index database. With adoption of the real-time indexing method based on the reverse index, on the premise of not affecting the searching performance as much as possible, the updated index database number is reduced, the limitation to the data size by the internal memory database is eliminated, a multipath processor is optimized in a targeted manner, data updating of a multipath concurrence multithreading is supported better, a more flexible data updating mode and efficient data updating efficiency are provided, the data timeliness is enhanced, and excellent searching experience is provided for a user.

Description

technical field [0001] The invention relates to an indexing method, in particular to a real-time indexing method based on an inverted index. Background technique [0002] For the field of text indexing, new data and updates can be performed without affecting the normal operation of the current system [0003] Real-time fast indexing of data has always been a subject of public relations. The real-time data update speed is very important especially for information services, which directly affects the user experience and the timeliness of push information. At present, the real-time update methods of the retrieval system are various, but they are all updated and optimized according to the index structure to speed up the update speed as much as possible. According to the index update method, it can basically be classified into two categories: [0004] (1) Main index library + auxiliary index library [0005] The main index library contains most of the data, and the auxiliary i...

AI Technical Summary

Problems solved by technology

[0008] Based on the method of main index library + auxiliary index library, in order to improve the update performance and shorten the update time, the auxiliary index library needs to be fully stored in the memory. As the amount of data increases in a single period of time, the auxiliary index library will cause memory A large amount of consumption must limit the amount of updated data in a single time, which greatly limits the application of this technology;

[0009] Based on the scheme of multiple disk index libraries + independent memory index library, after the capacity of the memory index library reaches a certain level, the memory library will be flushed to the disk, which avoids the capacity limit of the memory index, but due to the multiple disk index libraries Existence, and cannot be merged in time, will lead to the need to query multiple independent index libraries (multiple disk libraries + memory libraries) during the retrieval, which will reduce the retrieval performance and affect the user experience

Images