Method and system for collecting and storing stock data based on cloud computing technology

Abstract

The invention discloses a method for collecting and storing stock data based on a cloud computing technology. The method comprises the following steps: obtaining a plurality of stock codes; generating a corresponding URL (uniform resource locator) list file for each stock code; setting fragment size of a Hadoop distributed file system of a cloud computing platform; storing the URL list file into the Hadoop distributed file system; receiving a stock data collecting request submitted by a user; resolving the stock data collecting request into corresponding quantity of Map tasks according to the quantity of the file fragments; assigning the resolved Map tasks to a slave server on which corresponding fragments are located to carry out; transmitting an Http (hyper text transport protocol) data request to a remote stock data server from a server according to the resolved Map tasks; returning the stock data by the remote stock data server; and analyzing the stock data according to the resolved Map tasks, wherein compiling of analysis function program abides by structure description of a remote stock server on the stock data. The method is simple to develop and good in data structure expansibility, and rapid collection of the stock data can be achieved.

Description

technical field [0001] The invention belongs to the field of network application technology, and more specifically relates to a method and system for collecting and storing stock data based on cloud computing technology. Background technique [0002] The characteristics of the stock data generated on the securities trading day are mainly real-time and large-scale. At present, each stock in Shanghai and Shenzhen stock markets will generate an update data every 5 seconds during the trading day, and the daily data growth capacity is between 1.5-2GB. According to relevant statistical data in 2011, the number of listed and traded stocks on the Shanghai and Shenzhen Stock Exchanges in China is about 1,800, and that on the Hong Kong Stock Exchange is about 6,700. In foreign countries, the number of stocks listed and traded on the NASDAQ stock market in the United States is about 5,800. If funds and bonds are taken into account, the amount of data generated at the same time is even...

AI Technical Summary

Problems solved by technology

[0003] General database systems cannot meet such large-scale, real-time, and flexible requirements

There are problems such as complex development, poor real-time performance, difficult management, and poor flexibility of data structure in the implementation method of using relational database alone.

If the Oracle Timesten solution is adopted, there are problems such as high cost of software and hardware, difficulty in horizontal expansion, and the need for experienced technical staff

Images