Saturday, September 16, 2006

Issues of assessment

I had a very interesting day today shopping around for some stuff and it suddenly made me think of a very serious research area we need to look into here at DERI.

Looking back over today's adventures I realized that about 85% of the time, when I asked a question of a sales or service person here in Galway, the first answer I was given was incorrect.

These questions ranged from "is the item I ordered from Argos ready for pickup?" to "Do you have that in stock?", to "How many potato skins come in a small order?"

These were completely mundane questions and the sort of questions you would expect clerks, waiters, and service persons to know. However, in virtually every case, the FIRST answer given was incorrect. Analysis later (and some careful questioning of these folks) revealed that, largely, this was because they wanted to impart information, they wanted to be helpful, and in most case honestly believed the first answer was correct and were surprised to discover they were wrong. In one case the clerk was quoting a computer generated report and IT was wrong.

As I thought about this I suddenly realized that, if this is a cutltural phenomenon, it could be very important for issues of elearning, particularly assessment. In any culture where accuracy is sacrificed to answers, assessment of the efficacy of an elearning regime may need to take that into account to be effective and to properly reflect the state of the student.

So one very interesting research project would be to ask and answer the question,"How often is the FIRST answer to factual questions correct in a given culture/country/region?" Does it vary significantly with culture? Does it vary enough that it needs to be adjusted for in eLearning assessment strategies? Is this one more area where systems need to be contextually aware and make semantically powered adaptations?

Thoughts on this?