The invention discloses a
data transmission method based on logstash + kafka, which comprises operation of service data producing end and operation of service data consuming end. 1) automatically collecting the service data in the
service system database through the service data production terminal logstash, and transmitting the service data to the
middleware kafka; 2) the
middleware kafka collects the data transmitted by each service data producing end and uniformly sends the data to the service data consuming end logstash; 3) the service data consuming terminal logstash receives the messageof the
middleware kafka, converts the message into a
data file, and stores the message in the
target database. Compared with the prior art, the invention has the advantages of simple configuration andoperation, high
processing speed, and can effectively realize the operation of collecting,
processing, storing and the like of the dispersed service data, and can quickly and conveniently build the
transmission channel between the service data production end and the service data consumption end, and has good expansibility, and reduces the difficulty of development and operation and maintenance. The invention has the advantages of simple configuration and operation, high
processing speed, and effectively realizes the operations of collecting, processing, storing, etc.