A method and a web-based
software apparatus for use in the automated scoring of assessment test papers, utilizes both a human and the
machine scoring of each paper in a poly-metrological evaluation each assessment
score. The scoring performance of each human scorer, in web-base assessment scoring production, is constantly monitored and evaluated, in real time, for
score accuracy, bias, and other factors. Whereof, each human
score performance is measured against
machine score performance of the same assessment paper, and if need be, against a second human score performance in scoring the same assessment paper. Scores are resolved according to a subscriber approved
algorithm. Irresolvable discrepancies are addressed by a chief or master human scorer. The score performance history of each production, human scorer is constantly monitored, in real time, and each human scorer is prompted or selected-out for retraining, as necessary, according to a selected, real time,
evaluation algorithm. Scorer performance is judged according to exact agreement rates, and according to adjacent agreement rates.