IndexR real-time data analyzing library

Abstract

The invention discloses an IndexR real-time data analyzing library. The IndexR real-time data analyzing library realizes a structure data format which can be arranged in a distribution environment, can be subjected to parallel processing, is provided with an index and is of a rank type. On the basis of the data format, a data warehouse system is built through the IndexR; mass data sets can be subjected to rapid statistical analysis (OLAP) on the basis of Hadoop ecology; the data can be imported in real time, moreover, the zero delay of querying is realized. The IndexR is deigned to solve the problems such as slow analyzing under a big data scene, data delay, and complex system. According to the IndexR real-time data analyzing library, the data is stored in HDFS; the Zookeeper is used for communicating and negotiating in the cluster; the Hive is used for conveniently managing partitioned data; the data can be rapidly imported in real time through Kafka; an outstanding distributive querying engine Apache Drill is used in a querying layer.

Description

technical field [0001] The invention belongs to the technical field of the Internet, in particular to an IndexR real-time data analysis library. Background technique [0002] The programmatic advertising business needs to connect with major media on the entire network, and generate millions of analytical data per second. These data track and describe the process of advertising activities in detail, such as the number of creative impressions, clicks, registrations generated by activities, and return visits. We need to analyze and process these data in real time, including customer reporting, delivery optimization, fraud analysis, fee settlement, etc. The query pattern of data users is non-fixed and unpredictable, and with the surge of business volume, the amount of data also increases sharply. We need a new technology to solve these requirements: 1. Super large data set, low query latency: the query mode cannot be predicted and cannot be pre-calculated; the amount of table ...

AI Technical Summary

Problems solved by technology

4. Scalable, low cost, and easy maintenance: business will develop rapidly, new data sources will be generated, new tables will be added, and old data cannot be deleted, which will bring huge cost pressure and operation and maintenance pressure

The current Arquet, ORC and other data formats usually have good scanning and compression performance, but lack effective indexes and necessary flexibility

Images