67 results about "Distributed data flow" patented technology

The present invention is an improvement of methods and systems using mobile and distributed data stream mining algorithms for mining continuously generated data from different components of a vehicle. The system is designed for both on-board and remote mining and management of the data in order to (1) detect the effect of various engine parameters on fuel consumption behavior, (2) predictive classification of driving patterns and associative indexing of driver performance matrix, (3) resource-constrained anomaly detection for onboard health monitoring, (4) vehicle-to-vehicle social networking and distributed data mining, (5) adaptive placement of advertisements based on vehicle performance profile and (6) onboard emissions analytics computation for wireless emissions monitoring and smog test.

Embodiments of the present application relate to a distributed data stream processing method, a distributed data stream processing device, a computer program product for processing a raw data stream and a distributed data stream processing system. A distributed data stream processing method is provided. The method includes dividing a raw data stream into a real-time data stream and historical data streams, processing the real-time data stream and the historical data streams in parallel, separately generating respective results of the processing of the real-time data stream and the historical data streams, and integrating the generated processing results.

A system and method for synchronizing distributed data streams for automating real-time navigation through presentation slides in an electronic conference are provided. With the system and method, a participant in an electronic conference is permitted to navigate back and forth through a series of presentation slides while being able to automatically synchronize the participant's display to a current position within the series of presentation slides regardless of which slide the participant's computer is currently displaying. The present invention provides a background mechanism for monitoring the current live presentation by a presenter during an electronic conference while a participant navigates through a local copy of the presentation slides. Upon the selection of a synchronization option, the participant's display output of the presentation slides is synchronized with the current live presentation using the background mechanism.

Techniques for generating a distributed stream processing application are provided. The techniques include obtaining a declarative description of one or more data stream processing tasks from a graph of operators, wherein the declarative description expresses at least one stream processing task, generating one or more containers that encompass a combination of one or more stream processing operators, and generating one or more execution units from the declarative description of one or more data stream processing tasks, wherein the one or more execution units are deployable across one or more distributed computing nodes, and comprise a distributed data stream processing application binary.

The present invention relates to a system and method for performing vehicle onboard analysis on the data associated with the vehicle and implementing a cloud-based distributed data stream mining algorithm for detecting patterns from vehicle diagnostic and correlating the pattern with the contextual data. The system applies the distributed data mining algorithms for mining the results of the vehicle onboard analytics sent over the wireless network to the server and correlates the analyzed data with the contextual data of the vehicle. The system extracts performance patterns from data, builds predictive models from vehicle diagnostic, and correlates the predicted model with the business process using state of the art link analysis techniques.

The invention discloses distributed data stream clustering method and system and overcomes the defect that the existing most data steam clustering algorithms are unable to run in the distributed cloud environment, unable to easily extend and low in operational time efficiency. The method includes: summarizing data streams to obtain a plurality of eigenvectors of the data streams; performing locality-sensitive hashing algorithm to obtain a plurality of clusters with each comprising at least one eigenvector, and selecting at least one cluster as a candidate cluster; periodically using the candidate cluster to cluster eigenvectors of newly arrived data streams. The real-time performance better than that of the prior art is guaranteed by the use of the method and system based on the locality-sensitive hashing algorithm.

A computer readable storage medium includes executable instructions to assess system cache resources, inter-process communication requirements and staging requirements to divide an extract, transform, load (ETL) dataflow task into a plurality of sub-tasks. The sub-tasks are then executed in parallel on distributed resources.

The invention discloses a load balancing system, controller and method, and relates to the technical field of cloud computing. The system comprises the concentrated load balancing controller and load balancers, wherein the load balancing controller obtains topology information and data flow information of a load balancing cluster in the whole network, generates forwarding items for the load balancers according to a load balancing strategy, and sends the forwarding items to the scattered load balancers via a unified control interface; and the load balancers distributes data flows of the forwarding items to a background server. The load balancers scattered in the network are controlled in a centralized manner, and load balancing logics scattered in load balancing devices traditionally are brought into the concentrated load balancing controller in the unified manner, the problem that traditional load balancing is hard to manage in the unified manner is solved, the management difficulty is reduced, and the load balancing effect of the network is improved globally.

The invention discloses a distributed transmission method for a query data stream, which comprises the following steps of: 1) storing the data stream passing through each network node or the latest query result to form a view; 2) when a new query is generated in the network node, adopting the stored data stream passing through the network node or the latest query result in the transmission path as a new query data source, and adopting the network node including the new query data with least cost as a query execution node; and 3) querying the query data stream from a query generation node to the query execution node. The application of the method can realize shared transmission among a plurality of data streams as much as possible during distributed data stream query processing so that the data transmission quantity in the network is reduced.

Disclosed are an apparatus and a method for parallel processing continuous processing tasks in a distributed data stream processing system. A system for processing a distributed data stream according to an exemplary embodiment of the present invention includes a control node configured to determine whether a parallel processing of continuous processing tasks for an input data stream is required and if the parallel processing is required, instruct to divide the data stream and allocate the continuous processing tasks for processing the data streams to a plurality of distributed processing nodes, and a plurality of distributed processing nodes configured to divide the input data streams, allocate the divided data stream and the continuous processing tasks for processing the divided data streams, respectively, and combine the processing results, according to the instruction of the control node.

The invention provides a controlled host detection method and device based on a knowledge graph, and the method comprises the steps: filtering extracted feature data by adopting data, in a response state of NXDOMAIN, in DNS flow, describing the data based on a knowledge graph construction framework, constructing an NXDOMAINIP knowledge graph; and finally, analyzing the knowledge graph by utilizinga community discovery algorithm and a community judgment algorithm to obtain a controlled host list and mark a suspected malicious code family. Based on a distributed data flow processing framework,real-time DNS request flow is analyzed on a large scale, and the analysis efficiency is greatly improved through a multi-stage data preprocessing process; and through NXDOMAINIP knowledge graph construction, community discovery and community judgment, word list splicing DGA domain names are detected, controlled hosts and related malicious code families are determined, victims are reminded in timeto carry out AV upgrading and full-disk scanning on the hosts, and host vulnerabilities are reinforced.

The invention discloses a system and method for executing a combined service with centralized control flow and distributed data flow. The system comprises a central node and a plurality of service agents which are distributed on different nodes in a network. The invention has the advantages that: a control message for informing the service of when and where data is transmitted is sent, so that the problems of lack of monitoring capability of a distributed execution mode on the overall running situation of the combined service and easy existence of system performance bottleneck caused by the transition of data flow by a central node in a centralized execution mode can be solved. Simultaneously, overall execution logic of the combined service is controlled by adopting the central node, and the current business process execution language (BPEL) can be compatible, so that the execution of combined service application can be effectively supported.

The invention discloses a communication method and apparatus. The method includes steps: a first user plane function (UPF) network element establishes an auxiliary user plane channel to an auxiliary radio access network (RAN) network element; and the first UPF network element transmits received to-be-distributed downlink data transmitted by a data network to the auxiliary user plane channel according to to-be-distributed data flow information, wherein the to-be-distributed data flow information is saved in the first UPF network element and used for realizing data distribution in a 5G system. According to the method and apparatus, the utilization rate of radio resources is increased, the system switching time delay is reduced, and the user and system performance can be enhanced.

The invention relates to a Coflow collaborative job flow scheduling perception data flow division method and device, computer equipment and a storage medium. The method comprises the following steps:obtaining a to-be-divided DDSP application program data stream; extracting a complex calculation task flow of the DDSP application program; according to a preset Coflow collaborative job flow scheduling model, obtaining a Coflow collaborative job flow scheduling model, a complex calculation task flow in a DDSP application program is divided into a plurality of sub-task flows; further analyzing thedata access demand and the data dependence of each sub-task flow; carrying out multi-dimensional data segmentation on the DDSP application program data stream; reducing cross access to the data blocks among different tasks as much as possible and the dependence between the data blocks d; finally, according to the data access requirements of the sub-task flows in each computing node, taking data communication minimization between computing nodes and workload equalization of the computing nodes as optimization objectives, and allocating the segmented data blocks to the most appropriate computing nodes, so that the communication overhead between the distributed computing nodes is effectively reduced, the utilization rate of the data blocks is increased, the access speed is increased, and themethod is suitable for distributed data flow processing.

The invention provides a network data transmission method applied to transmission between communication devices. The network data transmission method comprises the steps of transmitting network data from a communication device which serves as a source terminal to a communication device which serves as a target terminal, and obtaining M first data transmission paths between the source terminal andthe target terminal; sequencing the M first data transmission paths according to the length of the transmission path from long to short, and then obtaining N second data transmission paths from the first data transmission paths in the sequence sequentially from low to high; distributing data flow corresponding to data transmission quality for each second data transmission path based on to-be-transmitted amount of network data; and transmitting the distributed data flow to the target terminal by each second data transmission path. Different transmission service demands are satisfied, and transmission efficiency of the network data is improved.

A system and method for synchronizing distributed data streams for automating real-time navigation through presentation slides in an electronic conference are provided. With the system and method, a participant in an electronic conference is permitted to navigate back and forth through a series of presentation slides while being able to automatically synchronize the participant's display to a current position within the series of presentation slides regardless of which slide the participant's computer is currently displaying. The present invention provides a background mechanism for monitoring the current live presentation by a presenter during an electronic conference while a participant navigates through a local copy of the presentation slides. Upon the selection of a synchronization option, the participant's display output of the presentation slides is synchronized with the current live presentation using the background mechanism.

The invention discloses a stochastic distributed data stream frequent item set mining system. In the system, a new distributed data stream mining mode for frequent item transactions is adopted, a stochastic frequent item mixer is introduced to improve the accuracy of mining, new transactions, when accumulated to the same number as basic windows, in a data steam are send to n frequent item set mining devices by a data item dividing method according to the different items in the transactions respectively, transaction numbers to which different data items belong are attached to the different data items, and in the frequent item set mining devices, frequent item sets are mined by running and operating different frequent item transactions and the mined frequent item sets are stored in a frequent item set storage. Finally, the data items are fed back to data item dividers through the stochastic frequent item mixer for deep mining. Compared with other methods, the method has the advantages of small memory storage space, high response speed and the like. Meanwhile, the coverage of mode mining can be ensured by increasing the number of the frequent item set mining devices or feedback times.

A system renders print jobs via a marking engine. A raster image processor receives the job from the input module and renders it to one or more raster images. One or more contone image processors receive the one or more raster images to perform cross-channel image processing to render the raster images into a form compatible with the marking engine, wherein the output is partially sub-partitioned by color separation. An image refactoring and distribution network completes the sub-partitioning by separation of the rendered contone raster images by color separation and distributes each color sub-partition to a marker channel module specific to each color sub-partition. A marking engine receives the color sub-partitioned image data, marks the image onto a substrate and outputs a hardcopy of the image.

The invention provides a continuous query method for K-Skyband on a distributed data stream. The continuous query method comprises the following steps of: receiving data transmitted by different data sources by a centre server, continuously reading the received data by a sliding window, then determining the result set of initial global K-Skyband points according to the data in the sliding window, taking the result set of the initial global K-Skyband points as the result set of current global K-Skyband points, carrying out overdue maintenance and updating maintenance on the result set of current global K-Skyband points according to the new data and overdue data generated after the sliding window reads the new data, and taking the result obtained after overdue maintenance and updating maintenance as a query result for outputting. Therefore, according to the method, the continuous query of the K-Skyband under a distributed data stream environment can be realized, and the application field of the K-Skyband continuous query method is expanded.

The invention provides a distributed database storage method, a system, a device and a storage medium. The method comprises the steps: obtaining data storage configuration data, wherein the data storage configuration data comprises the information of a synchronization table in a kafka message queue and the information of a landing table in a distributed database; receiving to-be-stored data from the synchronization table in the kafka message queue by a Flink distributed data flow engine; and inserting the to-be-stored data into a landing table of the distributed database by the Flink distributed data flow engine. By adopting the scheme of the invention, the time of a data processing link is extremely compressed. A multi-aggregation-node and multi-source service scene is supported. The useis humanized, and the data processing and analysis efficiency is greatly improved.

The invention provides a method, a system and an application for real-time evaluation of fund assets based on a distributed data stream. On the basis of horizontally expanding basic computing resources, the system and its application can stably and quickly import and process a large amount of fund asset data, which is a real-time calculation, and can meet the requirements of completing the valuation accounting of all fund assets within 3 seconds for 10,000 products, and simultaneously output the key index data such as net asset value, unit net value, paid-in capital, and so on. The invention solves the problem that the fund manager can not obtain the fluctuation of the fund net asset value in real time, and enables the user to obtain the real-time net asset value of the fund immediately after the fluctuation of the market and the investment of the fund manager. As that method and system of the invention are used, the investment personnel can timely and accurately perceive the investment result of the fund in the daytime, the fund manager can immediately perceive the change of the net value of the fund aft the investment, and can obtain the feedback net value of the fund. A method and system of that present invention.

In one embodiment, a method of monitoring a network. The method includes, at each node of a set, constructing a corresponding vector of M components based on a stream of data packets received at the node during a time period, the set including a plurality of nodes of the network, M being greater than 1; and estimating a value of a byte traffic produced by a part of the packets based on the constructed vectors, the part being the packets received by every node of the set. The constructing includes updating a component of the vector corresponding to one of the nodes in response to the one of the nodes receiving a data packet. The updating includes selecting a component of the vector to be updated by hashing a property of the received data packet.

In accordance with an embodiment, described herein is a system and method for capture of change data from a distributed data source system, for example a distributed database or a distributed data stream, and preparation of a canonical format output, for use with one or more heterogeneous targets, for example a database or message queue. The change data capture system can include support for features such as distributed source topology-awareness, initial load, deduplication, and recovery. A technical purpose of the systems and methods described herein includes determination and communication of changes performed to data at a distributed data source that includes a large amount of data across a plurality of nodes, to one or more target computer systems.

The present invention relates to the field of data processing, and particularly to a distributed method and device for processing a data stream, which are used for solving the problem that the distributed stream processing technology cannot conduct load balancing processing on data streams having the same key value. The method in the embodiments of the present invention comprises: according to a key value of a data stream which needs to be allocated to a next-hop work node, a first node determining a second node as the next-hop work node for processing a data stream corresponding to the key value, and after it is determined that the second node satisfies a set load migration condition, migrating the data stream, which needs to be allocated to the next-hop work node, to a third node for processing from the second node, and indicating that the second node and the third node are synchronizing state information about the data stream corresponding to the key value. By means of the method, after it is determined that a downstream second node satisfies a set load migration condition, a first node can migrate a data stream corresponding to the key value, which is mainly processed by the second node, to a third node for processing with a small accumulated load at present, so that load balancing processing can be conducted on data streams with the same key value.

Embodiments of the present invention include systems and methods for optimizing data flow in a network. The system for distributing data flow in a network includes a controller that receives, from a set of nodes coupled through the network, information of errors at the ports of each node through an input-output (IO) port. The controller compiles the information of errors to assign credits to links coupled to the ports; determines, based on the credits, how to distribute data flow in the network; generates a control signal for controlling the ports; and sends the control signal to the set of nodes through the IO port. The set of nodes controls the ports according to the control signal.

The embodiment of the invention discloses a terminal information processing and displaying method and system, and the method comprises the steps: collecting makeup trying experience data generated byan application end in a user makeup trying experience process, and transmitting the makeup trying experience data to an Flink stream calculation engine from the application end through a RocketMQ message queue; carrying out real-time stream processing operation on the makeup trying experience data through the Flink stream calculation engine, and abstracting the streaming makeup trying experience data into a distributed data stream; and storing the distributed data stream by adopting a cloud relational database, acquiring the distributed data stream which is processed by the Flink stream calculation engine in real time from the cloud relational database based on a Datav data visualization project, and displaying a statistical result through a display page. Data statistics and analysis are carried out on makeup trying experience of a user in an AI makeup trying physical store scene of the intelligent terminal, a result is displayed in real time, and a client can accurately and visually check experience conditions of current products and advertisements on the current day.

The invention discloses a local node mining method based on distributed data stream processing. The local node mining method comprises the following steps: S1, selecting a data sequence; S2, establishing a big data stream classification model; S3, a local node mining method; and S4, a center node mining method based on the unbalanced data stream. According to the local node mining method, the problem of local node mining in power grid equipment online monitoring data stream distributed processing is solved based on the data stream cleaning method for the power grid equipment from the perspective of large data stream distributed processing.

This specification discloses a web page downloading device with distributed data flows and the method thereof. When a browser sends out a web page request signal, the server end first uses a data flow monitoring module to detect the data flows of several transmission lines. A data flow distributing module then divides the multimedia contents, such as pictures, videos, executables, compressed files, etc, in the web page into several transmission packets. These packets are transmitted via the all the transmission lines.

A system and method for random sampling of distributed data, including distributed data streams. The system and method use a multi-level reservoir sampling technique that leverages the conventional reservoir sampling algorithm for distributed data or distributed data streams. The method establishes an intermediate reservoir for each distributed data source or data stream and populates the intermediate reservoirs with a sample of data elements received from each distributed data source or data stream. A final reservoir is established and data elements are randomly selected from each one of the intermediate reservoirs to populate the final reservoir.

The invention discloses a large-scale data flow analysis method and system based on Spark, and the method comprises the steps: constructing a distributed calculation framework based on Spark, and carrying out the distributed data flow analysis of a target code through the distributed calculation framework, wherein the distributed data flow analysis comprises an intra-process preprocessing stage and a distributed inter-process data integration stage. The system comprises a data acquisition module, a data processing module, a data analysis module, a data storage module and a data display module. According to the method and system, a two-stage parallel strategy is adopted, parallel computing is carried out in a multi-entry mode, high expandability is achieved, meanwhile, the load balancing problem of distributed computing is considered, the computing power advantage of a distributed cluster can be fully utilized, and static data flow analysis is accelerated.