The invention discloses a reference data access management method and device suitable for H.264 / AVC parallel encoding devices. The reference data access management device comprises a plurality of first-stage buffering units respectively connected with a coding core, the first-stage buffering units are connected to a second-stage buffering unit through a bus, and the second-stage buffering unit is further connected with an external storage device. Only when data accessed by the coding core does not exist in the first-stage buffering units and the second-stage buffering unit, the data are required to be obtained from the external storage device. According to the reference data access management method and device, the relativity of macro block multi-core parallel H.264 encoder to visit reference data can be sufficiently used, and the access amount of the external storage device to the reference data is sufficiently reduced. Repeated buffering block storage in each buffer memory is eliminated as much as possible, and the hardware cost of the reference data access management device is ultra low.