GIVE Generating Instructions in Virtual Environments

GIVE-2: Results

Seven NLG systems were evaluated in GIVE-2:

The evaluation contains both objective measures (success rate, completion time, etc.) and subjective measures, which were collected by asking the users to fill in a questionnaire.

The results, which have been presented at the INLG 2010, are available in the official GIVE-2 report.

Heatmaps

Heatmaps of each world-system pair of GIVE-2 are available for visualization of locations where players lost or cancelled the game.

Usability study

While the GIVE-2 Challenge was running, a team of researchers from Trinity College Dublin (Nikiforos Karamanis, Ielka van der Sluis, and Gavin Doherty) carried out a usability study on the GIVE-2 software. We plan to improve future installments of GIVE based on their findings. Their report can be found here.