The invention discloses a method for improving visual milemeter system robustness and reducing algorithm calculation consumption. The method comprises the following steps of 1, acquiring an image by means of the visual milemeter system through an image sensor, acquiring a frame p and initializing an MatchBufp to an empty set; 2, determining whether an MatchBufp-1 is empty, if yes, setting a camerapose Twc to 0, executing a step 6, and otherwise, executing a step 3; 3, calculating a relative pose Tlc between the frame p-1 and the frame p; 4, calculating a camera pose Twcp of the frame p in a map, and constructing the MatchBufp; 5, updating a FilterBuf, and determining the whether the frame p is a key frame; 6, constructing a new key frame Knew, and adding into SetK; and 7, initializing thenew three-dimensional point by means of characteristic matching. A real-time map construction and self-positioning system based on the image sensor has advantages of realizing accurate and robust composition and positioning, and reducing system calculation resource consumption.