Track C: Intelligent Interaction

Computers operate in everyday life as universal media machines presenting multi-media information and as communication devices connecting people. Interaction between the human user and the computer nowadays is a prominent topic of interest to researchers and designers. The design of intelligent interactive systems that allow users to interact naturally with the computer is the main focus of this track. The design of this new generation of advanced systems involves various issues from the field of human-computer interaction and artificial intelligence.
Several projects are possible.

1. Social robotics
As we get closer to having working social robots in our environments, we are starting to have a good idea how robots should move around in an environment that also contains humans. However, we are much further away from having unscripted interactions between humans and robots, where humans can communicate their desires and instructions to the robot naturally, using gestures and speech. Robots have limited and noisy sensing capabilities, and implementing such interaction patterns requires the robot to both track the humans' location and pose accurately, interpret their behaviour correctly and communicate its understanding and intentions effectively.
Projects taken up in this context will have access to a human-sized mobile robot platform, including basic localisation and motion planning capabilities, and the capacity to noisily track people around it. Below are a list of starting ideas for projects which you can take up or choose to extend. You can also come up with your own research questions and goals for a project.

  • Design and implement a gesture recognition system using the RGBD (kinect) camera, with a fixed vocabulary of gestures that can be used to direct the robot around in the environment. The system needs to recognise when the person is gesturing to direct the robot and not, and recognise the corresponding gestures.
  • Investigate how the robot should deal with situations when the person is not in view. In particular, if the instructions are given by a gesture sequence which forces the robot to lose track of the person, these instructions cannot be interpreted correctly and the robot should adjust its motion to take that into account.
  • The robot has a number of sensors that are very limited in their tracking abilities, such as laser-range finders at ankle level. Investigate how these inputs can be used for simple instructions (such as, in the limit, instructing the robot to focus other sensors on the person
  • Feedback: how should the robot communicate its understanding of the instructions, and how can we allow the user to correct mistakes?
  • Investigate how the robot should deal with multiple persons in a room.

2.  Question answer matching for structured interview simulation.
Interviews are dialogs in which participants play different roles. The interviewer asks questions, the interviewee is supposed to answer them. A simple method to simulate the interviewee by a machine is based on a collection of interviews. The question answer pairs are stored in a data base. When a question is asked the machine looks for the best matching question in his dbase and returns the associated answer from the dbase. This is the starting point for this assignment: a QA matcher and a dbase filled with QA-pairs extracted from real Dutch interviews. The existing QA matcher implements a simple implementation of ``matching''. The quality of the resulting system depends largely on the matching method. What goes wrong and how can the system be improved? Suppose we use spoken input instead of typed text as input. How robust is the system for recognition errors made by the (Dutch) speech recognizer?

3. Speech act and intention recognition.
There are different ways of questioning. "You did steal the money, didn't you!'' differs from  "Did you steal the money?'' and "What is your name?'' is different from "Your name please?''. Speakers do more when questioning than just asking for information. What different ways of questioning are there? Can we train machines to learn the different ways of questioning?

4. Building a machine learner for dialogue act type recognition in a natural language dialogue system.
This project builds on two earlier bachelor projects in which machine learners were developed for automatic dialogue act (in particular question type and content type ) classification. The aim of this project is to integrate these machine learners into an existing VR environment that allows a human to interact in Dutch with a virtual character that plays the role of a witness of a criminal event. For this implementation the API of the Data Mining Toolkit Weka can be used. Testing the system with non-expert users playing the role of a police interviewer is part of the project. Experience programming in Java is required.

5.Unethical computer games.
Designers of computer games design for user experiences. They try to influence the players' behaviour using game mechanics. The question is whether there is something like an (un)ethical game, ethical game design. Or are there only( im)moral game players? Or (im)moral designers? What are the characteristics of computer games that invite the player to become a more responsible actor in the game world. In general: what is the relation between morality and gaming? How do people/gamers see this? For a good start the students should read:  Miguel Sicart, The Ethics of Computer Games,  MIT Press, London, 2009.

6.Probabilistic Programming:  Exercises in Bayesian Inferencing and Machine Learning
Probabilistic programs are “usual” programs (written in languages like C, Java, LISP or ML) with two added constructs: (1) the ability to draw values at random from distributions, and (2) the ability to condition values of variables in a program via observe statements (which allow data from real world observations to be incorporated into a probabilistic program).

To get a quick idea of what PP is see the blog  The Programming Languages Enthousiast

It presents examples as well as links to programming languages and research papers.

The goal of probabilistic programming is to enable probabilistic modeling and machine learning to be accessible to the working programmer, who has sufficient domain expertise, but perhaps not enough expertise in probability theory or machine learning.We wish to hide the details of inference inside the compiler and runtime, and enable the programmer to express models using her domain expertise and dramatically increase the number of programmers who can benefit from probabilistic modeling.

What are you supposed to do in this research project?
- Get familiar with the idea of probabilistic programming: make some exercises. Make clear how probabilistic inference and machine learning can be encoded in a PP.
- Answer the following questions: does an introduction into probabilistic programming fit in a bachelor course in AI for computer science students?  If so what is the best language to use?

A good introduction in Probabilistic Programming is:

Andrew D. Gordon, Thomas A. Henzinger, Aditya V. Nori, and Sriram K. Rajamani. 2014. Probabilistic programming. In Proceedings of the on Future of Software Engineering (FOSE 2014). ACM, New York, NY, USA, 167-181. DOI:

See also the wiki pages:

or the book chapter:

Information about this assignment: Rieks op den Akker (

For further information on the content of this track, you may contact Herman Koppelman