Offline data acquisition method and device, equipment and storage medium

A collection method and collection device technology, applied in the field of data processing, can solve problems such as inconformity with database storage types, large manpower consumption, system crashes, etc., and achieve the effects of avoiding system crashes, reducing processing pressure, and meeting the needs of offline data collection

Active Publication Date: 2020-06-23
SUZHOU EAGLE NETWORK TECH CO LTD
View PDF8 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The inventors have found that there are at least the following problems in the prior art: the batch processing method adopted in the existing acquisition technology usually causes the system to crash due to the large amount of data, and the data format of the acquired offline data may not conform to reality The storage type of the database also needs to manually convert the acquired data into a data type that matches the actual database. Therefore, the existing offline data collection method not only has the risk of system crash but also requires a lot of manpower, which cannot meet the needs of users. Actual demand

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Offline data acquisition method and device, equipment and storage medium
  • Offline data acquisition method and device, equipment and storage medium
  • Offline data acquisition method and device, equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0027] figure 1 It is a flow chart of the offline data collection method provided by Embodiment 1 of the present invention. This embodiment is applicable to the case of collecting large-capacity offline data, and this method can be executed by the offline data collection device in the embodiment of the present invention. , the device may be implemented in software and / or hardware. Such as figure 1 As shown, the method specifically includes the following operations:

[0028] Step 101, split the offline data to obtain multiple offline data groups.

[0029] Wherein, each offline data group is smaller than the first preset capacity.

[0030] Optionally, before splitting the offline data to obtain multiple offline data groups, it may also include: determining that the offline data is greater than the second preset capacity, wherein the second preset capacity is greater than the first preset capacity; intercept.

[0031] Optionally, the data types in each offline data set are t...

Embodiment 2

[0044] figure 2 It is a flow chart of the offline data collection method provided by the embodiment of the present invention. This embodiment is based on the above-mentioned embodiments. In this embodiment, after each converted offline data group is merged to obtain the collected data, the Including: performing quality inspection on the collected data; loading the collected data into the target database when it is determined that the collected data meets the standards of the target database. Correspondingly, the method in this embodiment specifically includes the following operations:

[0045] Step 201, split the offline data to obtain multiple offline data groups.

[0046]Step 202, performing type conversion on each offline data set to obtain a converted offline data set.

[0047] Step 203, combining each converted offline data set to obtain collected data.

[0048] Step 204, performing quality inspection on the collected data.

[0049] Specifically, after merging each c...

Embodiment 3

[0057] image 3 It is a schematic structural diagram of an offline data collection device provided in Embodiment 3 of the present invention, and the device includes: an offline data acquisition module 301 , an offline data group conversion module 302 and a collection data acquisition module 303 .

[0058] Wherein, the offline data group obtaining module 301 is used to split the offline data to obtain multiple offline data groups, wherein each offline data group is smaller than the first preset capacity;

[0059] An offline data group conversion module 302, configured to perform type conversion on each offline data group to obtain a converted offline data group, wherein the converted offline data group meets the storage type requirements of the target database;

[0060] The collected data acquisition module 303 is configured to combine each converted offline data group to obtain collected data.

[0061] The above-mentioned device can execute the off-line data collection method...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses an offline data acquisition method and device, equipment and a storage medium. The method comprises the steps that offline data are split to obtain multiple offline data sets, and each offline data set is smaller than a first preset capacity; performing type conversion on each offline data set to obtain a converted offline data set, the converted offline data set meeting the storage type requirement of the target database; and combining each converted offline data set to obtain acquired data. According to the embodiment of the invention, the high-capacity offline data is split; the data processing pressure can be reduced; according to the offline data acquisition method and device, system crash caused by too large processed data volume is avoided,each split offline data set is converted into the data meeting the storage type requirement of the target database, developers do not need to repeatedly encode for conversion, and manpower consumptionis reduced, so that the requirement of users for offline data acquisition is met.

Description

technical field [0001] Embodiments of the present invention relate to the technical field of data processing, and in particular, to a method, device, device, and storage medium for collecting offline data. Background technique [0002] At present, the industry has accumulated a large amount of data, and the volume, type and change of the data are increasing sharply. When processing data, in order to improve the speed of data processing, batch processing is usually used for processing. [0003] The inventors have found that there are at least the following problems in the prior art: the batch processing method adopted in the existing acquisition technology usually causes the system to crash due to the large amount of data, and the data format of the acquired offline data may not conform to reality The storage type of the database also needs to manually convert the acquired data into a data type that matches the actual database. Therefore, the existing offline data collection ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25
CPCG06F16/258
Inventor 宋大伟丁静
Owner SUZHOU EAGLE NETWORK TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap