A Method of Dynamically Computing News Gathering Service Resources

Abstract

The invention discloses a method for dynamically calculating news collection service resources. The method is based on the news data collected in the past and the amount of resources invested in collecting the data, and extracts the features of the data, and determines the frequency of data collection for a specific website through dynamic analysis of a logistic regression model, and then dynamically determines the frequency of data collection for a specific website. The collection resources required for data collection, and then the actual collected data volume and resource input are used as feedback information to continuously modify the parameters of the logistic regression model, and dynamically correct and optimize the collection frequency. Through the method, the collection frequency and resource input can be dynamically adjusted and optimized during the collection process, effectively overcoming the problems of missed collection and high collection cost, and greatly reducing the collection cost on the premise of ensuring the collection quality.

Description

technical field [0001] The invention belongs to the technical field of data analysis, and in particular relates to a method for dynamically calculating news collection service resources. Background technique [0002] News websites update data frequently every day, and there are a large number of sites. For enterprises engaged in website data mining and analysis, a large number of server / bandwidth / IP resources are required to collect data resources of news websites. The use of each type of resource will There are substantial costs involved. If the collection frequency of news websites is too low, it is easy to miss the collection of news; if the collection frequency is high, the cost of server / bandwidth is high. [0003] Existing collection systems generally collect website data resources at a single frequency. Some excellent collection systems use hierarchical management to simply classify websites, and collect data resources at a fixed frequency for each category. With th...

AI Technical Summary

Problems solved by technology

[0011] In order to solve the problems that the existing data collection system statically sets the data collection frequency, which leads to incomplete data collection, omission of important data, or waste of computing, storage and network resources, resulting in high collection costs, etc., the present invention provides a dynamic computing news collection service resource According to the method, the method is based on historical data, extracts features from the data, and determines the data collection frequency of a specific website through dynamic analysis of a logistic regression model, and then dynamically determines the collection resources required for data collection of a specific website, Such as computing, storage, network resources, etc.