GIVE-2: Results

Seven NLG systems were evaluated in GIVE-2:

• one system from the Complutense University of Madrid, Spain (Madrid);
• one system from the Dublin Institute of Technology, Ireland (DIT);
• one system from the University of Heidelberg, Germany (Heidelberg);
• two systems from INRIA Grand-Est, France (NA and NM);
• one system from Saarland University, Germany (Saar);
• one system from Trinity College Dublin, Ireland (TCD).

The evaluation contains both objective measures (success rate, completion time, etc.) and subjective measures, which were collected by asking the users to fill in a questionnaire.

The results, which will be presented at INLG 2010, are available in the official GIVE-2 report.

Heatmaps

Heatmaps of each world-system pair of GIVE-2 are available for visualization of locations where players lost or cancelled the game.

Usability study

While the GIVE-2 Challenge was running, a team of researchers from Trinity College Dublin (Nikiforos Karamanis, Ielka van der Sluis, and Gavin Doherty) carried out a usability study on the GIVE-2 software. We plan to improve future installments of GIVE based on their findings. Their report can be found here.