311 results about "Data scraping" patented technology

A hybrid prediction system may aggregate electronic data to identify and initially predict an outcome of a future event and subsequently update the initial prediction. The system may include at least one processor and a memory. The processor may access data scraped from the Internet. The data may be associated with at least one future event. The processor may further store the scraped data, determine, from the scraped data, an initial prediction of the outcome of the at least one future event, generate, from the scraped data, an initial likelihood indication associated with the initial prediction, and transmit the initial prediction and the initial likelihood indication to a device associated with one or more users. The processor may further receive proprietary information, store the proprietary information, determine, using the scraped data and the proprietary information, a subsequent likelihood indication, and transmit the subsequent likelihood indication to the device.

The system and method facilitate the control and management of information and actions related to the computerized scanning, categorization, processing, storage, and retrieval of paper documents and electronic document images for the purposes of creating electronic data; data entry; data capture; data classification and categorization; and document-, file-, and records-management. The systems and methods facilitate the conversion of paper documents into electronic images and into more useful forms.

Embodiments of the present invention provide methods and system for allowing an open overlay service for a social network to interface with other online services. In particular, the open overlay service configures its clients to make requests on behalf of the social network. Responses to the requests are routed back to the open overlay service and the information is then provided to the social network. Since the requests come from the client the online services will simply treat them as normal requests, and thus, the open overlay service can seamlessly integrate with a user's online services.

The invention provides a commodity comment data labeling system based on hierarchical AP clustering. The system includes a data capturing module, a word vector training module, a feature information extraction module and a feature information labeling module. The data capturing module stores corpus information and comment data. The word vector training module obtains a training corpus set. The feature information extraction module obtains a feature information set corresponding to the comment data. The feature information labeling module obtains the comment data labelled result after clustering. The beneficial effects of the invention are as follows: the commodity comment data labeling system and method based on hierarchical AP clustering are provided, the purpose of automatically labelingcomment data is achieved, the value orientation of the feature information can be mined and presented in the form of labels to merchants and customers, support is provided for subsequent data analysis, and companies and consumers are further provided with a convenient, scientific and intuitive tool for obtaining useful comment information.

The invention discloses a method and a system for fetching business data. The method includes configuring rule data required during fetching operation; reading the rule data, creating web page resource fetching tasks according to the rule data, and storing fetched web page resources according to configuration rules in a classified manner; creating data analysis tasks for the fetched web page resources, analyzing HTML (hypertext markup language) documents for the fetched web page resources to acquire required resource URIs (uniform resource identifiers) and filtering out the resources with incomplete data; creating resource download tasks, and downloading the resource URIs acquired by means of analysis in a breakpoint resume manner to acquire resource data; storing the resource data or fetching other resource data according to the integrity of the resource data and transmitting reporting information if the fetching operation cannot be completed normally. The method and the system have the advantage that problems that large quantities of resources are consumed when data are acquired while business data cannot be acquired by means of configuring relevant information are solved.

The invention provides a system and method for constructing a level knowledge base based on inference. The system comprises a data capturing module, an Ontology module, a knowledge extraction module and a knowledge construction module, the Ontology module is used for constructing and updating field Ontology, classification Ontology and overall Ontology, the knowledge extraction module is used for extracting attribute content in each data source to obtain needed field knowledge, the attribute content can be mapped into the field Ontology, the knowledge construction module comprises a knowledge fusion sub-module and an inference engine sub-module, the knowledge fusion sub-module is used for fusing and storing the field knowledge from all the data sources in the same field to obtain a basic knowledge base, the inference engine sub-module is used for calling the Ontology module and defining the classification Ontology, related inference rules are made, the inference rules are applied to the basic knowledge base, and potential knowledge is obtained and stored in classification knowledge bases corresponding to all classifications. The system and method can facilitate the maintenance and update of the knowledge base.

The invention discloses a product database forming method based on Internet data and a system. The method includes the steps of capturing webpage data with the theme relevance higher than a preset threshold value by the adoption of the focused crawler technology, performing structuralized storage on the captured webpage data, automatically classifying the structuralized storage webpage data according to the categories which products belong to, performing statistics on the frequency and the time of occurrence of attributes of the products in the webpage data after the automatic classification, performing weighting calculation on the frequency and the time of occurrence of the attributes of the products according to preset weighting, acquiring the decision value of the attributes of the products, and determining the sort order of the attributes of the products according to the decision value of the attributes of the products. The system comprises a data capturing module, a structuralized storage module, a data classifying module and an attribute deciding module. According to the product database forming method based on the Internet data and the system, a user can acquire comprehensive and summarized information without needing to collect and sort product information on the Internet, real-time performance of data is ensured, and real-time requirements of the user are met.

The invention relates to a data collection method based on an intelligent grasping system for science and technology service information. The method comprises the following steps: 1, data grasping: in terms of configuration of a crawler, a user releases a grasping task through a configuration module and a start module of a client and sets a webpage for grasping and corresponding rules; 2, loading of timing grasping tasks: the task released by the user is loaded dynamically to a timing grasping task list; 3, page downloading; 4, page parsing: pages in a queue are parsed; 5, adding to URL to be grasped; 6, data processing and storage: page data is subjected to parsing extraction processing, and extracted two-dimensional structured data is stored. The method can meet the requirement of the crawler for generality and the requirement of the science and technology service system for grasping, and convenient extension and plug-in type development are realized; parser rule configuration, width and depth of the pages subjected to grasping, grasping threads, database configuration or index configuration is added to specific service logic, and information can be grasped and collected intelligently.

The invention provides a method for processing data among systems by using Excel. the method comprises the following steps: performing various configurable data capturing on an Excel file exported out of a source system according to a parameter table rule by using a configuration parameter table; converting codes of the data according to a coding rule; and verifying the data through a verification rule. Therefore, the processing on data of a source system to become data in a target system format can be realized, the collecting workload of massively similar data can be reduced, and the data quality can be further improved effectively and quickly.

The invention particularly discloses a single-image robot disordered target grabbing method based on pose estimation and correction. The method comprises the steps: S1, generating an image data set ofa to-be-grabbed object model; S2, constructing a convolutional neural network model according to the image data set in the step S1; S3, importing the two-dimensional image of the to-be-grabbed objectinto the trained convolutional neural network model to extract a corresponding confidence map and a vector field; S4, obtaining a predicted translation amount and a predicted rotation amount of the to-be-grabbed object; S5, finding the optimal grabbing point of the object to be grabbed and calculating the measurement translation amount of the depth camera; S6, performing grabbing safety distancecorrection according to the predicted translation amount of the object to be grabbed and the measured translation amount of the depth camera, executing correction data grabbing if the correction succeeds, and entering S7 if the correction fails; and S7, repeating the steps S3-S6. The disordered target grabbing method provided by the invention has the characteristics of high reliability, strong robustness and good real-time performance, can meet the existing industrial production requirements, and has a relatively high application value.

The invention discloses a commodity networked gene based brand intellectual property protection platform comprising a data source module, a data collection module, a data integration module, a data storage module, a data analysis module, an object detection module, a visual module and a data application module. When the data collection module collects data source data, an open-source Hadoop platform is utilized for constructing a distributive whole-network commodity data capturing system; the data analysis module is utilized for structured arrangement of a great number of unstructured commodity comment data; the object detection module is utilized for analysis and detection of suspected infringing commodities by the aid of an established infringing commodity identification model; the visual module is utilized for displaying the analyzed and detected suspected infringing commodities via a visual interface. The commodity networked gene based brand intellectual property protection platform has the advantages that fewer human and material resources can be utilized, a larger-scale market can be handled effectively, and intellectual property maintenance cost is reduced for enterprises, so that economic benefits are increased.

The invention discloses a recommendation method based on a knowledge-aware hypergraph neural network, which comprises the following steps: step 1, constructing a user hypergraph, and initializing a hyperedge by taking an article having an interaction relationship with a user as an entity node; constructing an article hypergraph, wherein the initial hyperedge takes an article having an interaction relationship with any user having the interaction relationship with the article as an entity node; step 2, convolution calculation is performed on the user hypergraph and each article hypergraph; and step 3, calculating the inner product of the user and all the articles to obtain the interaction score of the user and all the articles. The method has the beneficial effect that the auxiliary information of the articles is integrated into the vector representation of the user, so that the articles of which the user is more likely to have the interaction relationship can be selected from the numerous articles. The invention discloses a recommendation system based on a knowledge-aware hypergraph neural network. The recommendation system comprises: a data capture module; a knowledge perception hypergraph construction module; a domain convolution module; a hyperedge convolution module; and a prediction module. The method has the beneficial effects of simple operation and accurate prediction.

The embodiment of the invention discloses a computer robot crawling task distribution method and device, and a computer robot data crawling method and device. The computer robot data crawling method comprises the following steps: a crawled historical page is stored into historical crawling data, whether a historical page corresponding to the URL(Uniform Resource Locator) cluster of a crawling task is in the presence in the historical crawling data or not is judged when the crawling task is in the presence, first target page data can be directly extracted from an available historical page if the historical page is in the presence and the existing historical page is available, the part of the historical page does not need to be repeatedly crawled, and system resources are saved. Meanwhile, the network bandwidth usage rates of all computer robots are detected in fixed time, the mean value E(w) and the variance D(w) of the network bandwidth usage rate of each computer robot are calculated and stored, and then, the availability of each computer robot is calculated according to E(w) and D(w). For the available computer robots, the computer robot for executing tasks is selected according to the descending order of an availability probability so as to reasonably distribute computer robot resources.

The invention discloses a user data processing system which comprises a user data capture module, a data integration processing module, a quantitative classification statistics module, a residence track analysis module, a store resource database, a store resource updating module, a management server and a guide push initiation module. The user data capturing terminal is respectively connected withthe quantitative classification statistics module and the data integration processing module; wherein the quantitative classification statistics module is connected with the comprehensive quantitative evaluation module through the resident trajectory analysis module, the commodity resource database is connected with the comprehensive quantitative evaluation module, and the management server is respectively connected with the data integration processing module, the comprehensive quantitative evaluation module, the store resource database and the guide push initiation module. According to the invention, the user position information and the consumption record are analyzed to determine the preference and interest of the user, so that the guidance information can be accurately pushed to guidethe user to shop, the satisfaction degree of the user is greatly improved, and the sales of commodities in the shopping mall is promoted.

The invention provides a data grabbing method and a data grabbing system. The data grabbing method comprises the steps that a plurality of regular expressions are configured; the regular expressions are selected from the regular expressions in sequence according to the preset sequence of the regular expressions, data correlated with target data are matched in a target file according to the selected regular expressions, if the correlated data are matched, the correlated data are grabbed and returned, the selection of the regulation expressions from the regular expressions is terminated, the matching operation in the target file is terminated, and if the correlated data are not matched according to the regular expressions, prompt information is returned. By means of the technical scheme, when one system needs to obtain data of other systems, corresponding data can be obtained from other systems in a matching mode only by configuring the corresponding regular expressions, and data sharing can be conveniently carried out between the different systems.

The invention provides a data fetching method and device of an OTT (over the top) application. The data fetching method comprises the following steps: indicating a to-be-tested terminal to fetch signaling layer data generated when the OTT application sends a signaling in business of the OTT application arranged in the to-be-tested terminal, wherein the signaling layer data comprise signaling information of a signaling sent every time and time information corresponding to signaling sent every time; fetching application layer data generated by the OTT application from the data output end of a simulative network device, wherein the application layer data comprise information of an application data packet sent every time and time information corresponding to the application data packet sent every time; and relating the signaling information of the signaling layer data matched with the time information with the application data packet information of the application layer data to form behavior data corresponding to the OTT application. By utilizing the data fetching method and device, the technical problem that the synchronous fetching of the signaling layer data and network layer data can not be realized so as not to guide an operator to optimize network behaviors of the OTT application in the prior art is effectively solved.

The invention discloses a visualized analysis method based on industry data. The visualized analysis method includes capturing the industry data on the internet, performing integrated analysis on the industry data and industry internal organization data, performing internet data arrangement, and by the aid of the R-language analysis technology and front-end display charts, displaying the data in terms of different dimensionalities according to industry requirements. Compared with the prior art, the visualized analysis method based on the industry data has the advantages of capabilities of classifying the data effectively, filtering useless data and expressing forms clearly and high analysis speed, and the problem about data match due to the fact that in spite of desiring to know dynamic market change, improve service quality, increase goal achievement rate and the like, existing enterprises in many industries cannot display and analyze the internet data through visualized interfaces and cannot combine the internet data with own industry internal organization data is solved.

The invention discloses a data processing device which comprises a data capturing module for capturing data according to a pre-configured capturing rule; and a data processing module for processing data captured by the data capturing module according to a pre-configured data conversion rule to obtain standard data which meet the data conversion rule. The invention further discloses a data processing method without manually screening and processing data, so that the time cost of manually screening and processing data is greatly saved, the work efficiency is improved, the manpower and material resources are saved, and the data processing accuracy can be further improved.

The application discloses an HTML template-based method for releasing graphics and text information via a television. The method comprises the steps as follows: setting a template file which is suitable for HTML / xHTML format played by the television; wherein the template file comprises a display frame and a formal parameter, and the display frame defines a display mode of metadata; the formal parameter is embedded in a specific position of the display frame; using a data capturing technology to capture the metadata from a webpage and storing the metadata in a database; extracting the metadata from the database and filling in the template file to replace the formal parameter, and rendering and generating a graphics and text page; converting the graphics and text page into a picture format; converting the graphics and text page in picture format into a PAL video signal of a television program channel, and playing on a television screen. The application further discloses an HTML template-based system for releasing the graphics and text information via the television, and a content releasing server.

The present invention provides a data capture method and system. The method comprises the following steps of receiving a data range of a user needing to capture; according to the data range, carrying out the data capture by a Baidu search algorithm and a Google search algorithm separately; taking the same results in the Baidu search algorithm and the Google search algorithm as the capture results. The technical scheme provided by the present invention has an advantage of good capture effect.

A server computer system for parsing non-uniformly presented data from a variety of unique non-uniform third-party web portals can comprise a scripting processor configured to automatically execute a web-portal specific script for each of the one or more third-party web portals accessed by a network communication device. Each of the web-portal specific scripts can be configured to imitate inputs from a user input device and to automatically adapt interactions with each of the one or more third-party web portals to access and parse data elements from one or more non-uniformly available data fields. Further, the server computer system can comprise a database processor configured to compare a first set of data received from the one or more non-uniformly available data fields with a second set of data, which is stored within the local database device.

The invention discloses a behavior analysis method for construction personnel of a power substation engineering construction project, and relates to an intelligent video system based on a machine learning method and a chromatographic partition management technology. A computer network communication technology, a video monitoring technology and a safe area management technology are adopted to analyze and process construction site monitoring data. Function division and safety attributes of a regional grid working plane are set, color identification is performed on regions, real-time monitoring is performed on environment, facility and equipment state changes, threshold early warning is set, binding interaction is performed on personnel and the identified regions and equipment, and data capture and classification are performed on related information. According to the invention, real-time monitoring can be realized, abnormal conditions can be judged, and an alarm can be given at the fastest speed and in the maximum mode, so that beforehand early warning can be effectively carried out.

The invention provides a UHV DC transmission line tour inspection and feedback system based on big data, and belongs to the field of transmission line tour inspection for transmission line maintenance. The UHV DC transmission line tour inspection and feedback system comprises a data storage server, a node server, an Internet data capture server, a mobile data capture server, and a grid load monitoring server. The data storage server comprises a storage unit and a processing unit. The storage unit is used for storing data extracted by the node server, the Internet data capture server, the mobile data capture server and the grid load monitoring server. The information is stored by the processing unit. In view of the above technical scheme, the UHV DC transmission line tour inspection and feedback system can realize the information linkage between a UHV DC transmission line node and an associated electricity consumption area through a plurality of ways so as to ensure real-time dynamic monitoring of the transmission line with less manpower input, thereby timely eliminating hidden troubles.

The invention relates to the technical field of data analysis and acquisition, in particular to a webpage data capture method. Data information of some websites with access right is captured quickly and effectively by means of establishing a data channel of concurrent execution and defining a data capture process of the websites. Facing to ERP (enterprise resource planning) software developers, a scheme for quickly conveniently defining data capture of corresponding websites is provided, and trouble of artificially accessing to website downloaded information is avoided by timed automatic data capture of a background.

The invention relates to a data capture system. The system comprises a task duplicate removal module, a task queue module, a task scheduling module, a data capture module and a result queue module. The invention furthermore relates to a data capture method. The method comprises the steps of receiving data capture tasks sent by business lines and performing duplicate removal; forming a task queue by the tasks subjected to the duplicate removal; calculating a task priority based on a double polling algorithm, scheduling the tasks based on the priority, and allocating the tasks to crawler nodes; capturing data in the internet by utilizing a crawler; and returning the captured data, forming a result queue and sending the result queue to the business lines.

The invention discloses a database system applied to a real-time big data scene. The database system applied to the real-time big data scene is characterized by comprising a data capturing module which is used for defining the range for capturing a data source, automatically capturing webpage data, performing content extraction and repetition exclusion, and performing context analysis; a feature database module which is used for storing Cookie data, advertisement position dada and linked data; a real-time database module which is used for performing indexing and sharding storage on the data in the feature database module; an advertisement putting module which is used for obtaining advertisement feature data from the real-time database module, comparing the advertisement feature data with the data in the feature database module, and finally determining whether to put advertisement to specific users and which advertisement is put to the users. The database system disclosed by the invention has the beneficial effects that as a three-level storage structure of internal memory, SSD and hard disk is adopted, the system not only provides high-speed real-time data access ability, but also can effectively cope with super-large scale data storage and has the disaster recovery ability.

The embodiment of the invention provides a webpage data capturing method and device, a storage medium and equipment. The method comprises the steps that after a target webpage is successfully logged in through a headless browser, if the headless browser monitors an AJAX request of the target webpage, authorization authentication information carried in the monitored AJAX request is stored in a cache through the headless browser; the authorization authentication information is read from the cache through the data capture script, and the read authorization authentication information is added intoan access request for capturing webpage data; and webpage data returned by the server is captured after passing the authentication based on the authorization authentication information through the data capture script and based on the access request containing the authorization authentication information. Therefore, the webpage data can be effectively captured.