The invention provides a magnetoencephalography system and a method of operating the magnetoencephalography system, and relates to the technical field of medical imaging. The magnetoencephalography system comprises a flexible brain magnetic cap having inserting grooves, and a limiting clamping sleeve which cooperates with the inserting grooves and comprises a fixing part, the fixing part is arranged on the surface, adjacent to the inserting grooves, of the limiting clamping sleeve, and the limiting clamping sleeve is used for fixing an atom magnetometer into the inserting grooves. According to the magnetoencephalography system provided by the invention, by virtue of the flexible brain magnetic cap with the inserting grooves, the limiting clamping sleeve cooperating with the inserting grooves and linkage matching between the limiting clamping sleeve and the fixing part on the adjacent surfaces of the inserting grooves, the purpose of portable detection of brain magnetic signals of a subject is achieved. The flexible brain magnetic cap provided by the invention can adapt to head contours of different sizes within a certain range, the portability of magnetoencephalography detection is improved, and the wearing experience is greatly improved.