The present invention is a method and system for segmenting a plurality of persons in a physical space based on automatic behavior analysis of the persons in a preferred embodiment. The behavior analysis can comprise a path analysis as one of the characterization methods. The present invention applies segmentation criteria to the output of the video-based behavior analysis and assigns segmentation label to each of the persons during a predefined window of time. In addition to the behavioral characteristics, the present invention can also utilize other types of visual characterization, such as demographic analysis, or additional input sources, such as sales data, to segment the plurality of persons in another exemplary embodiment. The present invention captures a plurality of input images of the persons in the physical space by a plurality of means for capturing images. The present invention processes the plurality of input images in order to understand the behavioral characteristics, such as shopping behavior, of the persons for the segmentation purpose. The processes are based on a novel usage of a plurality of computer vision technologies to analyze the visual characterization of the persons from the plurality of input images. The physical space may be a retail space, and the persons may be customers in the retail space.