Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Online shop big data processing method and device, equipment and storage medium

A big data processing and data technology, applied in the field of data processing, can solve the problems of partition scan failure, slow execution speed, slow data reading, etc., and achieve the effect of reducing reading failure, scheduling commodity data volume stably, and improving scheduling speed.

Active Publication Date: 2021-11-19
北京值得买科技股份有限公司
View PDF15 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] With the accumulation of business data, the amount of e-commerce product data is getting larger and larger, and the reading of data is getting slower and slower, resulting in a slowdown of the entire data processing process
Taking the Hbase database as an example, the Hbase database uses partitions as data units. There can be many pieces of data in a partition, and 4096 partitions can store up to 200 million commodity data. Use a Python program to traverse each partition sequentially and read commodity data Finally, do scheduling, distribution, download and analysis operations. The problem with this method is: the execution speed is slow, and when scheduling a round of data, it takes 1.5 hours to 2 hours; and, during the execution process, there will be partition scanning failures , resulting in data that cannot normally enter the downstream process

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Online shop big data processing method and device, equipment and storage medium
  • Online shop big data processing method and device, equipment and storage medium
  • Online shop big data processing method and device, equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] Preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the invention are shown in the drawings, it should be understood that the invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0037] The terminology used in the present invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein and in the appended claims, the singular forms "a", "the", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations o...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to an online shop big data processing method and device, equipment and a storage medium. The method comprises the following steps: reading commodity data in an Hbase through a Spark program, and generating an elastic distributed data set RDD; converting the RDD into the DataFrame; filtering data in the DataFrame through an online shop website ID and shelve-unshelve conditions; converting the DataFrame with data filtered into the RDD again; matching each piece of data of the reconverted RDD with a pre-configured acquisition scheme and an acquisition task through a MapPartitions operation of the RDD, and generating an optimal scheduling item; and pushing the generated optimal scheduling item into different link pools according to the task frequency. According to the technical scheme provided by the invention, the scheduling speed of the commodity data is greatly improved, the volume of the scheduled commodity data is stable, and the situation of reading failure of partition data is greatly reduced.

Description

technical field [0001] The present invention relates to the field of data processing, in particular to an e-commerce big data processing method, device, equipment and storage medium. Background technique [0002] With the accumulation of business data, the amount of e-commerce commodity data is getting larger and larger, and the reading of data is getting slower and slower, resulting in a slowdown of the entire data processing process. Taking the Hbase database as an example, the Hbase database uses partitions as the data unit. There can be many pieces of data in a partition, and 4096 partitions can store up to 200 million commodity data. Use a Python program to traverse each partition sequentially and read commodity data Finally, do scheduling, distribution, download and analysis operations. The problem with this method is: the execution speed is slow, and when scheduling a round of data, it takes 1.5 hours to 2 hours; and, during the execution process, there will be failur...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F16/25G06F16/2457G06F9/54
CPCG06F16/278G06F16/258G06F16/2457G06F9/546G06F2209/548
Inventor 隋国栋高景洋刘峰刘超
Owner 北京值得买科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com