Tobii Sticky mean validation error explained
The mean validation error is a key metric used to evaluate the accuracy of gaze tracking predictions against known ground truth data. Using this value to classify sessions ensures that the accuracy aligns with established research benchmarks.
Consider every frame from the ‘validation’ phase where ground truth is known.
Compute the difference between predicted gaze and ground truth (also known as L1 distance measure) along both the axes, take the max value, and finally compute the mean validation error, as shown below.
Mean validation error = mean (Error per frame)
pred_x = predicted gaze point on x-axis
pred_y = predicted gaze point on y-axis
gt_x = ground truth on x-axis
gt_y = ground truth on y-axis
Mean validation error criteria per category
The quality classifications described below (high, moderate, and low) help in understanding and categorizing the performance of gaze tracking systems.
High quality
The mean validation error must be within the 0 – 0.1800 range to be classified in this category.
Threshold value: 0.1800 is chosen as the upper limit as per State-of-the-art (SOTA) research conducted (https://arxiv.org/abs/1606.05814) on gaze tracking. The underlying principle is based on what this threshold limit translates regarding validation error in distance (cm).
By setting the threshold as 0.1800, the mean validation error is 3.02 cm along the max dimension, which aligns with the SOTA research.
It is worth mentioning that 3.02 cm was computed with an assumed mean resolution of (1462 x 733) and 166 Pixels Per Inch (PPI) was taken from historical data. Resolution and PPI do not have an absolute value since they depend on the device used. Therefore, 3.02 cm doesn’t mean it would be the same for all devices but rather was used as a placeholder for the assumption.
Moderate quality
The mean validation error must be within the 0.1800 – 0.2455 range to be classified in this category.
By setting the upper limit to 0.2455, the mean validation error was 4.08 cm, a delta of 1.06 cm from that of high-quality sessions.
Considering high-quality sessions, logging blinks and extreme head poses throughout the study, and using statistical models to fit the data helped understand their behavior. Setting the upper limit at 0.2455 resulted in a 25% standard deviation difference in the mean number of blinks and extreme head poses compared to high-quality sessions.
Therefore, sessions within the 0.1800 – 0.2455 range can be included along with the high-quality ones if necessary.
Low quality
Any session with a mean validation error greater than 0.2455 is classified in this category.
