Saturday, September 16, 2006

Issues of assessment

I had a very interesting day today shopping around for some stuff and it suddenly made me think of a very serious research area we need to look into here at DERI.

Looking back over today's adventures I realized that about 85% of the time, when I asked a question of a sales or service person here in Galway, the first answer I was given was incorrect.

These questions ranged from "is the item I ordered from Argos ready for pickup?" to "Do you have that in stock?", to "How many potato skins come in a small order?"

These were completely mundane questions and the sort of questions you would expect clerks, waiters, and service persons to know. However, in virtually every case, the FIRST answer given was incorrect. Analysis later (and some careful questioning of these folks) revealed that, largely, this was because they wanted to impart information, they wanted to be helpful, and in most case honestly believed the first answer was correct and were surprised to discover they were wrong. In one case the clerk was quoting a computer generated report and IT was wrong.

As I thought about this I suddenly realized that, if this is a cutltural phenomenon, it could be very important for issues of elearning, particularly assessment. In any culture where accuracy is sacrificed to answers, assessment of the efficacy of an elearning regime may need to take that into account to be effective and to properly reflect the state of the student.

So one very interesting research project would be to ask and answer the question,"How often is the FIRST answer to factual questions correct in a given culture/country/region?" Does it vary significantly with culture? Does it vary enough that it needs to be adjusted for in eLearning assessment strategies? Is this one more area where systems need to be contextually aware and make semantically powered adaptations?

Thoughts on this?

Bill

2 comments:

Anonymous said...

I think you are right - I did not thought about that at first - but now it seems that event in our institute we kind of applied this technique of questions answering.

I remember one of my (senior) colleagues told me the other day that very often instead of coming up with an answer based on analyzing the questions - he was quickly delivering short answer that he thought was the correct one.

Another example might be that we analyzed the way some of people handle requests - by saying "I will ask ...", "I have to consult with ...", "Interesting idea, let me think about that", and so on. And we thought that it could be a great plugin for email client for people that are getting to much mail - just to pretend they read the email, send immediate response, and deffer the actual actions to some time in the future.

However, you have to take into consideration that this behavior is probably related to the type of work one does - one example - politicians - they never answer questions directly.

Anonymous said...

That is interesting. I would imagine that the rate at which a person responds correctly to a question the 1st time would be associated with how that question is 1st interpreted by the brain.

Take for instance the problem of a response. How many times has someone asked a question, you respond with a "what?" and then just before they ask their question again, you spurt out the answer to that question.