ICT304 NLP Foundation for Generative AI End-of-Course Assessment 2026, Singapore

ICT304 End-of-Course Assessment

This assignment is worth 60% of the final mark for ICT233, Data Programming.

(full marks: 100)

Question 1

ECA Overview

In this ECA, you are tasked with building a customer-support bot. Your goal is to design and implement key components of a dialogue system that can accurately understand user intents and track dialogue states in a multi-domain customer service environment. This project will challenge your understanding of Natural Language Processing (NLP) concepts, machine learning model implementation, and practical application development.

MultiWOZ-2.2 Dialogue Acts Dataset:

• This dataset comprises approximately 8,000 multi-domain customer-support dialogues.
• Each user turn within these dialogues is annotated with; Intent: The user’s primary goal (e.g., “BookTaxi,” “FindRestaurant,” “Inform”); (Slot, Value) Pairs: Specific pieces of information the user provides or asks about (e.g., (destination, “Cambridge”), (food, “Italian”), (price range, “cheap”)).
• The dataset is structured into standard splits: train.json, for training your models; dev.json, for validation, hyperparameter tuning, and early performance evaluation; and test.json, for local evaluation and initial performance reporting.

Test-Blind.json

• This file follows the same schema as test.json but has all labels (intents and slot-value pairs) hidden. This will be used for the final evaluation on the Kaggle leaderboard to ensure unbiased assessment. Will be released nearing the submission date.

Evaluation Script:

• A provided Python script (evaluate.py) will compute the following standard metrics for dialogue state tracking (will be provided during class):
  • Intent Accuracy: The proportion of user turns where the predicted intent exactly matches the gold-standard intent.
  • Slot Micro-F1: A micro-averaged F1 score across all slot-value pairs, measuring the precision and recall of individual slot predictions.
• You can run this script locally on your test.json predictions to monitor your progress. The same script will be used to evaluate your submissions on Test-Blind.json for the leaderboard.

Question 1a

Your first task is to thoroughly explore and describe the provided MultiWOZ-2.2 dataset. This foundational understanding is crucial for effective model design.

Data Exploration:

• Analyze the domain balance: What are the primary domains covered (e.g., taxi, restaurant, hotel, train, attraction)? How are dialogues distributed across these domains?
• Identify the slot vocabulary: List unique slot names across all domains. Quantify the size of the slot vocabulary and describe the types of values associated with common slots.
• Calculate the average utterance length (in words or tokens) for user and system turns.
• Examine the dialogue distribution (in turns).
• Describe the ontology of the dataset, detailing how intents and slots are structured within and across domains. You may consider visualizing parts of the ontology (e.g., a tree/diagram).

(15 marks)

Question 2

Simple Model Experimentation for Slot Filling

This section focuses on building and evaluating models for intent recognition and slot filling, starting with simpler baselines and progressing to a more integrated approach. Ensure you use the train.json for training and dev.json for evaluation unless otherwise specified.

Question 2a

Vanilla DistilBERT + K-means for Domain Prediction Only

• Task: Design a model that predicts only the domain label for each user turn.
• Architecture: Utilize a pre-trained DistilBERT model (e.g., distilbert-base-uncased from Hugging Face Transformers) to extract embeddings for each user utterance. Apply a K-means clustering algorithm on these embeddings, where the number of clusters (k) is set to the total number of unique domains. Map these clusters to domain labels.
• Evaluation: Report the accuracy of your domain predictions on the dev.json split.
• Report: Illustrate your implementation steps, the chosen DistilBERT model, the K-means configuration, and the observed domain accuracy.

(10 marks)

Question 2b

Intent prediction

• Repeat the steps in part Q2a to predict the active_intent label.

(15 marks)

Question 3

Question 3a

Joint Prediction of Intent and Slots:

• Task: Extend your model to jointly predict both the intent and all associated (slot, value) pairs for each user turn.
• Approach: Fine-tune DistilBERT for a multi-task learning setup. This typically involves applying separate classification heads on top of DistilBERT’s pooled output: one for intent classification and another for slot tagging.
• Evaluation: Report both the Intent Accuracy and Slot Micro-F1 on the dev.json split using the provided evaluation script.
• Report: Discuss the architecture modifications, training methodology (e.g., loss functions, optimizer, number of epochs), and the resulting performance metrics.

(20 marks)

Question 3b

Error Analysis

Based on your model’s performance on the dev.json data, identify and discuss TWO (2) typical error patterns you observe in its predictions (e.g., common misclassifications, issues with specific slot types, or failures in multi-turn contexts). Provide concrete examples from the dev.json dataset to illustrate these errors.

(5 marks)

Question 4

Test set Submission

• Task: Use your “best model” (the one achieving the highest combined performance on dev.json from to predict the domain and slot-value pairs for each user turn in the Test-Blind.json file.
• Submission: Format your predictions according to the specified Kaggle competition format (details will be announced in class, typically a JSON or CSV file). Submit your predictions to the dedicated Kaggle competition leaderboard.
• Evaluation: Your final score will be determined by your rank on the Kaggle leaderboard based on the official evaluation metrics (Intent Accuracy and Slot Micro-F1) on Test-Blind.json.

(20 marks)

Question 5

Dialogue systems are highly domain-dependent. This question asks you to evaluate the current system and fine-tune your approach for other scenarios.

• Task: The MultiWOZ-2.2 dataset has predefined slots and intents primarily for tasks like booking taxis or making restaurant reservations. Your task is to propose how to adapt the ontology of this customer support bot for a completely new scenario.
• New Scenario: Choose ONE (1) of the following:
  • E-commerce Product Support: A bot assisting customers with inquiries about product features, order status, returns, or troubleshooting for an online retail store.
  • Healthcare Appointment Scheduling: A bot assisting patients with booking, rescheduling, or canceling appointments with various medical specialists.
• Ontology Adaptation: For your chosen scenario:
  • Propose at least FIVE (5) new primary intents relevant to the scenario.
  • For each new intent, identify and define at least THREE (3) relevant slots (with example slot values where applicable).
  • Describe how the dialogue flow might differ from the MultiWOZ domains based on your new ontology.
  • Briefly discuss potential challenges in data collection and annotation for your new scenario.

(15 marks)

———– End of ECA Paper ————-