The invention discloses a vision sensing-based bridge structure health detection system, which comprises a binocular vision bridge detection module, an additional detection sensor group composed of astrain sensor, an accelerometer, a temperature sensor, an air velocity transducer, a vehicle detector and the like, a data wireless communication module, and a data processing cloud platform, whereinthe binocular vision bridge detection module dynamically tracks a detection target point through a binocular camera by using a template matching algorithm, and a displacement of a bridge target pointunder three-dimensional coordinates is calculated by combining a three-dimensional reconstruction algorithm, so as to detect the flexibility of the bridge target point; and meanwhile, size detection and classification can be further carried out on a crack on the outer side of a bridge by using an image acquired in real time; multiple data detected by the additional sensor group can correct and supplement binocular vision sensing detection parameters, so as to realize overall dynamic analysis for the health of a bridge structure; and real-time health detection and early warning of the bridge structure are realized through wireless communication and the cloud platform. The whole detection system has the advantages of easy installation, quick and real time measurement, high accuracy, high stability and the like, and is applicable to various different bridges and easy to popularize.