Tracks
DSTC8 has the following four tracks (please click the links to get more detailed information about each track):
Multi-domain Task Completion
This track offers two tasks to foster progress in two important aspects of dialog systems: dialog complexity and scaling to new domains.
Task 1 – We provide a comprehensive platform for handling multi-domain end-to-end task completion dialogs in the travel domain. Participants are free to work on any type of components of a dialog system such as NLU, policy, NLG or end-to-end neural model as long as it aims to improve performance in end-to-end settings.
Task 2 – Participants leverage a large scale open-domain dialogue dataset to predict user responses in goal-oriented dialogues by learning how to quickly adapt a response model. Models are given few dialogues from a held-out domain to adapt before predicting responses on held-out test dialogues of that same domain.
NOESIS II: Predicting Responses
This task explores three dialogue challenges: next utterance selection, task success, and conversation disentanglement. In the primary task, participants must predict the next utterance in a conversation out of a set of 100 options, which in some cases will not contain the correct answer at all. Two datasets will be used, one on student advising and a new, higher quality set of dialogues extracted from the Ubuntu IRC help channel. Additional subtasks consider (1) next utterance selection when given raw IRC channel content as input, (2) identifying success in student advising, and (3) disentangling conversations in the Ubuntu IRC channel.
Audio Visual Scene-Aware Dialog
This track is a follow-up challenge of DSTC7 for Audio Visual Scene-Aware Dialog (AVSD) using Natural Language Generation (NLG) technologies. The task is to build a system that generates response sentences in a dialog about an input video. All combination of Dialog, Question Answering, Video scene understanding technologies can be tested.
Schema-Guided State Tracking
Virtual assistants need to support an ever-increasing number of services and APIs. This track explores challenges associated with dialogue state tracking in such a setting. Each dialogue in the dataset is accompanied by schemas listing a set of user intents and slots, and their natural language description. The dialogue state needs to be predicted over these intents and slots. Submissions will be judged based on the ability to generalize to new schemas which are possibly not present in the training set, both in single domain and multi domain dialogues.