Using Spoken Dialog Systems to Assess L2 Learners’ Oral Skills in a Local Language Testing Context

Abstract

The assessment of oral proficiency in instructional contexts presents interesting opportunities when designed to foster communicative performance in a specific target language use domain. A task-based Tourism English oral performance assessment was designed using a specialized spoken dialogue system (SDS) in which the computer is programmed to act as a hotel guest and examinees respond as a hotel employee. Based on a mixed-method research design, this study examined whether task administration conditions and the rubric for scoring performance assessment are appropriate for providing evidence of Tourism English ability. The analysis of 30 L2 students’ oral performances, their post-test questionnaire responses, and semi-structured individual interviews indicates that they considered the tasks engaging and relevant to their future profession. Similarly, four raters’ verbal reports from semi-structured individual interviews suggest that the test tasks effectively elicited ratable speech samples that can be argued to represent students’ oral communication skills in a hospitality setting.

Appendices

Appendices

1.1 Appendix A

1.1.1 Sample Transcript of a Test Task

• 1 c Hi, I am calling from room 263. It is very hot in my room, the A/C is not working.

• 2 u Oh I’m sorry to hear that how can I help you?

• 2 c You are not hel** me.

• 3 u Okay I can change your room if you’d like.

• 3 c That won’t work for me. Just do something about the A/C.

• 4 u Okay I’m.. I am sending a technician to fix the problem.

• 4 c You should have considered getting this fixed before you assigned me to this room.

• 5 u Okay. Sorry. I can give you a refund if you like.

• 5 c Giving my money back won’t fix the problem.

• 6 u I know. But I am trying to find a solution.

• 6 c Look, I really appreciate you trying to help me. Is there a chance you can suggest a reasonable solution?

• 7 u Okay. Then I can move you to a new room.

• 7 c That’s too much work for me. My luggage is very heavy.

• 8 u All right. Let me see.

• 8 c Are we still talking about the A/C?

• 9 u Yes.

• 9 c I don’t think you are even trying to help me.

• 10 u if you don’t want to move, then I can.. I can send a technician to fix the problem but you don’t want it either.

• 10 c Alright, if you say so.

Note: c = Computer, U = user.

1.2 Appendix B

1.2.1 Rating Scale

Skills	Rating
	4	3	2	1
Interactional competence	Task performed competently with almost always appropriate responses given in each task	Task performed generally competently with usually appropriate responses given in each task	Task performed somewhat competently and with somewhat appropriate responses given in each task	Task not completed due to limitations in responses given in each task
Fluency	Appropriate speech rate (i.e., no unnatural language -related pauses)	Mostly appropriate speech rate (i.e., few unnatural language -related pauses)	Somehow appropriate speech rate (i.e., some unnatural language -related pauses)	Inappropriate speech rate (i.e., too many unnatural language -related pauses)
Pronunciation	Very effective pronunciation and prosodic patterns with only minimal errors in production	Above average pronunciation and prosodic patterns but with occasional errors in production interpretation	Somehow average range of pronunciation and prosodic patterns but with errors in production affecting the delivery	Limited range of pronunciation and prosodic patterns with errors in production highly affecting the delivery
Grammar/vocabulary	A high degree of grammatical accuracy in both simple and complex structures and domain-specific vocabulary use	Sufficient grammatical accuracy in both simple and complex structures and domain-specific vocabulary use.	Somehow sufficient grammatical accuracy in both simple and complex structures and domain-specific vocabulary use	Insufficient grammatical accuracy in both simple and complex structures and limited domain-specific vocabulary use.

1.3 Appendix C

1.3.1 Test Takers’ Questionnaire Responses in ‘Overall’ and ‘Preference’ Categories

Category	Questionnaire item	1*	2	3	4	5
Overall	1. I really enjoyed the tasks.	0.0%	0.0%	9.5%	61.9%	28.6%
	2. The topic was relevant to real-life hotel situations.	4.8%	4.8%	9.5%	14.3%	66.7%
	3. If I know I’ll have a speaking test like this, I might study to improve my speaking.	0.0%	0.0%	19.0%	33.3%	47.6%
Authenticity	4. I think the interaction I had with the computer was natural.	0.0%	4.8%	14.3%	28.6%	52.4%
	5. The computer did not allow me to think about what to say.	0.0%	9.5%	42.9%	33.3%	14.3%
	6. I needed to respond quickly as in real life telephone conversation with a customer.	0.0%	4.8%	9.5%	33.3%	52.4%
	7. There were instances when the computer did not respond to what I said.	0.0%	0.0%	19.0%	47.6%	33.3%
Preference	8. I would prefer to discuss with a human partner next time.	23.8%	57.1%	4.8%	9.5%	4.8%
	9. I would prefer to discuss with a computer partner next time.	4.8%	14.3%	19.0%	28.6%	33.3%
	10. I’d like to see more tasks like this in the future.	4.8%	4.8%	28.6%	33.3%	28.6%
Self-evaluation	11. The test allowed me to demonstrate my oral communication skills at a hotel situation.	0.0%	0.0%	28.6%	38.1%	33.3%
	12. The computer understood me well.	0.0%	4.8%	38.1%	47.6%	9.5%
	13. I was able to understand the computer well.	4.8%	9.5%	28.6%	28.6%	28.6%
	14. The difficulty of the task was appropriate for me.	0.0%	19.0%	19.0%	42.9%	19.0%
	15. The computer’s speech was fast for me.	0.0%	4.8%	9.5%	19.0%	66.7%

1. *1=Strongly Disagree, 2=Disagree, 3=Neutral, 4= Agree, 5= Strongly Agree