Beyond test accuracy: Benefits of measuring response time in computerised testing

Abstract

This paper investigates how monitoring the time spent on a question in a test of basic mathematics skills can provide insights into learning processes, the quality of test takers' knowledge, and cognitive demands and performance of test items that otherwise would remain undiscovered if the usual test outcome of accuracy only format ("correct/incorrect") was used. Data from three tests of basic mathematics skills taken by tertiary students in 2004-2006 were analysed. Means and distributions of individual response times on parallel test questions were examined and differences were further investigated.

Analysis of response time data revealed a number of surprising findings in regard to the impact of variables on preferences for written and mental calculation methods and regarding additional cognitive demands of a question. The study examined how simple statistical analysis of response time distribution can be used to investigate abnormalities of the item functioning. These findings may be of value to educators and to test producers by informing them about the potential of utilising response time measurements as a diagnostic facility in computerised tests, for the purposes of improving teaching and learning.