Character string value domain segmenting method and device

Abstract

The invention discloses a character string value domain segmenting method and a character string value domain segmenting device which are applied to the field of data warehouse integration. The method comprises the following steps: extracting character strings with the maximal ASCII (American Standard Code for Information Interchange) code value and the minimal ASCII code value in a data primary key to be extracted; converting the character strings into big integers by weight expansion sum according to the ASCII code values and a preset scale cardinal number to form a range to be segmented; solving a range difference; performing equal segmenting on the range to be segmented according to the range difference and a preset segmenting number to obtain a segmenting step length so as to obtain big integers corresponding to segmenting nodes; reducing the big integers corresponding to the segmenting nodes into segmenting node character strings by adopting an Euclidean algorithm; generating a plurality of data extracting statements according to the segmenting node character strings to realize multithreaded accelerated extracting. The method is also optimized, namely, a reduced scale cardinal number is adopted in the processes of the weight expansion sum and the Euclidean algorithm. According to the method and the device, the concurrency and the high efficiency in a data transmission process are greatly improved.

Description

Technical field [0001] The invention is applied to the field of data warehouse integration, and relates to a string value domain segmentation method and device, and specifically relates to an application in the field of data warehouse integration for offline data extraction, equalizing numbers and strings according to the value domain range Split to achieve multi-threaded accelerated extraction method. Background technique [0002] In the era of big data, data needs to be continuously flowed and exchanged in order to maximize its value. In the construction of enterprise data warehouses and business intelligence, it is usually stored in various RDBMS (relational databases, such as Mysql, Oracle, PostgreSQL, etc.) The online data is synchronously extracted to offline storage and computing platforms for unified processing (such as Hadoop in the open source community, ODPS and other systems within Alibaba Group), such as figure 1 As shown, online data will also be migrated to other o...

AI Technical Summary

Problems solved by technology

[0004] But this approach has some drawbacks

[0005] First, convert the string to a very small number (such as BigDecimal) representation, but in the conversion operation of the string, an exception may be thrown, resulting in segmentation failure

[0006] Secondly, in order to avoid exceptions that may be thrown during the conversion operation of the string, it is often necessary to perform approximate processing on the converted data (such as rounding, etc.) when selecting an adaptive algorithm, so that the precision is damaged and the string cannot be restored accurately. In addition, when some extremely small numbers are restored to strings, the maximum length of restored characters must be specified. This limitation also leads to the inability to restore precision

[0007] Furthermore, in the process of converting an alphabetic string to a very small number, 65536 is used as the base, and this range basically accommodates all European and American characters and most Asian characters, and then restores the ASCII string after segmentation During the process, non-ASCII characters may also appear

[0008] In addition, for different types of strings to be split, the method of converting to extremely small numbers is not universal, and it is not applicable to any strings, such as integer types / time types in RDBMS, etc.

Images