Evaluation of students’ Noncognitive Large Scale Assessments measures: Problem Solving Experiences scale as a case study

• Hind Hammouri Department of Educational Psychology, The Hashemite University, Zarqa, Jordan
• Mutasem Mohammad Akour Department of Curricula and Instruction, The Hashemite University, Zarqa, Jordan
• Saed Sabah Department of Educational Psychology, The Hashemite University, Zarqa, 13115, Po Box 150459. Jordan

Palabras clave: Diverse cultures, Program evaluation, Scale Model, Student

Resumen

This study evaluated students’ Noncognitive Large Scale Assessments NLSA measures by examining the psychometric properties of the Problem Solving Experiences (PSE) scale as a case study of this kind of measures. We analysed the responses of 2000 students participated in PISA2012 from Germany, Japan, Jordan, and the United States of America to the personality scales of openness and perseverance (student questionnaire) that were assessed under the construct PSE. Results of the Rasch Rating Scale Model RSM analysis revealed that the structure of the PSE-scale differs between countries. There are problems of category disordering. 10% of items demonstrated misfit to the Rasch model. There are differences and inconsistency in items’ endorsability by country. The scale is not unidimensional and lacks sufficient sensitivity to discriminate individuals with high levels from those with lower levels of PSE. Accordingly, this study provided evidence that it is difficult to adopt the same noncognitive measure in different cultures in order to provide data that enables making decisions regarding intended purposes. Results of this kind of measures should be handled cautiously.