Data processing method and system
A data processing system and data processing technology, applied in the field of communications, can solve the problems affecting the data throughput performance and low data throughput performance of the Storm system, and achieve the effects of improving data throughput performance, reducing pressure, and saving computing resources
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0073] This embodiment records a data processing method, which can be applied to a data processing system (such as a Storm system), such as Figure 6a As shown, the data processing method recorded in this embodiment includes the following steps:
[0074] Step 101, the data processing system acquires business data from a data source according to preset data source configuration information.
[0075] As an example, the component Spout in the Storm system can obtain business data from external data sources.
[0076] Step 102, distribute the acquired business data to the N data processing layers of the data processing system.
[0077] N is the number of different granularities indicated by the service processing policy corresponding to the service data, the granularity is a basic unit for processing the service data to obtain service information, and N is an integer greater than 1.
[0078] Step 103, at the i-th processing layer of the data processing system, combine the receive...
Embodiment 2
[0095] This embodiment records a Storm system, such as Figure 7 As shown, the data processing system includes:
[0096] The data source unit (corresponding to the component Spout in the Storm system) is used to obtain business data from the data source according to the preset data source configuration information;
[0097] The data source unit is further configured to distribute the acquired business data to N data processing layers of the data processing system, where N is the number of different granularity businesses indicated by the business processing policy corresponding to the business data, and the Granularity is the basic unit for processing the business data to obtain business information, and N is an integer greater than 1;
[0098] The data merging unit (corresponding to the component Boltb1 in the Storm system) is used to merge the received business data at the i-th granularity at the i-th processing layer of the data processing system to obtain the i-th granula...
Embodiment 3
[0114]The Storm system recorded in this embodiment can support the access of multiple business data, that is, it can process the business data of multiple data sources accessed, avoiding the problem of low Bolt code reuse rate in related technologies, and can Save computing resources of the Storm cluster; such as Figure 8a As shown, the Storm system recorded in this embodiment includes:
[0115] Spout is used to transmit data of at least two services to the Bolt. The data is transferred between the Spout and the Bolt in the form of a stream, and the basic transmission unit of the stream is a tuple;
[0116] Bolta is used to clean data from multiple businesses, including filling data with default values, aligning special values, judging legality, etc., filtering out unnecessary data, that is, “dirty data” according to Bolta’s own business logic " process, the processed data is sent to Boltb in the form of a stream (the basic unit is a tuple);
[0117] Boltb is used for norma...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com