The invention relates to the technical field of image processing, in particular to a 3D skeleton modeling and hand detecting method. The method includes the steps that a figure video is shot through a depth video camera; face detection is carried out in the video; based on face depth information, a body silhouette is extracted; the body silhouette is verified, whether the silhouette is a body silhouette is judged, if yes, the next step is executed, and if not, the face detection step is returned to be executed; image filtering and smoothing are conducted; a skeleton line is extracted by using a detailing algorithm; the distance of body parts is calculated, and a probability distribution diagram is established; all the parts of a human body are recognized; precise joint points are obtained, and coordinates of hands in the 3D space are also obtained; all the joint points are connected to form a complete 3D human skeleton; the 3D human skeleton and the coordinates of the hands are output; the body silhouette is tracked, and information is provided for the next frame. According to the 3D skeleton modeling and hand detecting method, people can be detected rapidly under an existing technical condition, a human skeleton model is obtained, each joint point of people can be accurately positioned, the 3D skeleton is established, the arithmetic speed is high, the computation complexity is low, the 3D skeleton modeling and hand detecting method is suitable for various complicated backgrounds, and only 5ms is needed for each image frame.