A virtual control system for substantially real-time imaging machines, such as, for example, ultrasound, is presented. In exemplary embodiments of the present invention, a virtual control system comprises a physical interface communicably connected to a scanner / imager, such as, for example, an ultrasound machine. The scanner / imager has, or is communicably connected to, a processor that controls the display of, and user interaction with, a virtual control interface. In operation, a user can interact with the virtual control interface by physically interacting with the physical interface. In exemplary embodiments according to the present invention the physical interface can comprise a handheld tool and a stationary tablet-like device. In exemplary embodiments according to the present invention the control system can further include a 3D tracking device that can track both an ultrasound probe as well as a handheld physical interface tool. In such exemplary embodiments a user can control scan and display functions of the ultrasound machine by moving a handheld tool relative to the stationary tablet, and can perform 3D interactive display and image processing operations on a displayed 3D image by manipulating the handheld tool within a defined 3D space. Alternatively, all control functions, those associated with scan and display control as well as those associated with 3D interactive display and image processing can be mapped to manipulations of the handheld tool in a defined 3D space.