In the claimed approach, a high efficiency video coding codec optimizes the memory resources used during motion vector (MV) prediction. As the codec processes block of pixels, known as coding units (CUs), the codec performs read and write operations on a fixed-sized neighbor union buffer representing the MVs associated with processed CUs. In operation, for each CU, the codec determines the indices at which proximally-located “neighbor” MVs are stored within the neighbor union buffer. The codec then uses these neighbor MVs to compute new MVs. Subsequently, the codec deterministically updates the neighbor union buffer—replacing irrelevant MVs with those new MVs that are useful for computing the MVs of unprocessed CUs. By contrast, many conventional codecs not only redundantly store MVs, but also retain irrelevant MVs. Consequently, the codec reduces memory usage and memory operations compared to conventional codecs, thereby decreasing power consumption and improving codec efficiency.