A method for dynamically calculating news gathering service resources

Abstract

The invention discloses a method for dynamically calculating news collection service resources. The method is based on the news data collected in the past and the amount of collection resources invested for the collection data, extracts the feature of data, dynamically analyszes by a logistic regression model to determine the frequency of data collection for specific websites, then dynamically determines the collection resources required for data collection of specific websites, and then through the actual data collection and resource input as feedback information, constantly amends the parameters of the logistic regression model, and dynamically revises and optimizes the collection frequency. The method can dynamically adjust and optimize the acquisition frequency and the resource input in the acquisition process, effectively overcomes the problems of leakage mining and high acquisition cost, and greatly reduces the acquisition cost under the premise of ensuring the acquisition quality.

Description

technical field [0001] The invention belongs to the technical field of data analysis, and in particular relates to a method for dynamically calculating news collection service resources. Background technique [0002] News websites update data frequently every day, and there are a large number of sites. For enterprises engaged in website data mining and analysis, a large number of server / bandwidth / IP resources are required to collect data resources of news websites. The use of each type of resource will There are substantial costs involved. If the collection frequency of news websites is too low, it is easy to miss the collection of news; if the collection frequency is high, the cost of server / bandwidth is high. [0003] Existing collection systems generally collect website data resources at a single frequency. Some excellent collection systems use hierarchical management to simply classify websites, and collect data resources at a fixed frequency for each category. With th...

AI Technical Summary

Problems solved by technology

[0011] In order to solve the problems that the existing data collection system statically sets the data collection frequency, which leads to incomplete data collection, omission of important data, or waste of computing, storage and network resources, resulting in high collection costs, etc., the present invention provides a dynamic computing news collection service resource According to the method, the method is based on historical data, extracts features from the data, and determines the data collection frequency of a specific website through dynamic analysis of a logistic regression model, and then dynamically determines the collection resources required for data collection of a specific website, Such as computing, storage, network resources, etc.