Workshop Programme
Click here for a PDF version of the schedule. Talks are 10 minutes long, with 5 minutes for questions.
08:45 – 09:30 – Welcome and Introduction by Workshop Organizers
- Keith Miller and Stacey Bailey on Automated and Task-Based Evaluation of the Effects of Machine Translation Domain Tuning on MT Quality, Post-editing, and Human Translation Slides
- Lucia Specia and Kim Harris on the MQM framework Slides
- Comments from Alan Melby and Tyler Snow on Understanding Stakeholder Requirements for Determining Translation Quality MQM Slides and Survey Slides
09:30 – 10:30 – Talks Session 1
- Joke Daems, Lieve Macken and Sonia Vandepitte, Two Sides of the Same Coin: Assessing Translation Quality in Two Steps Through Adequacy and Acceptability Error Analysis Slides
- Leonid Glazychev, How to Reliably Measure Something That’s Not Completely Objective: A Clear, Working and Universal Approach to Measuring Language Quality Slides
- Mihaela Vela, Anne-Kathrin Schumann and Andrea Wurm, Translation Evaluation and Coverage by Automatic Scores Slides
- Arle Lommel, Maja Popović and Aljoscha Burchardt, Assessing Inter-Annotator Agreement for Translation Error Annotation Slides
10:30 – 11:00 – Coffee break
11:00 – 13:00 – Talks Session 2
- Marianne Starlander, TURKOISE: A Mechanical Turk-based Tailor-made Metric for Spoken Language Translation Systems in the Medical Domain Slides
- Caitlin Christianson, Bonnie Dorr, and Joseph Olive, MADCAT Evaluation Approach: Operational Accuracy of MT applied to OCR Slides
- Ekaterina Stambolieva, Continuous Operational Evaluation of Evolving Proprietary MT Solution’s Translation Adequacy Slides
- Lars Ahrenberg, Chunk Accuracy: A Simple, Flexible Metric for Translation Quality Slides
- Michael Carl and Moritz Schaeffer, Word Transition Entropy as an Indicator for Expected Machine Translation Quality Slides
- Doug Jones and Tamas Marius, A New Multiple Choice DLPT-STAR Comprehension Test for MT and Standardized ILR-Based and Task-Based Speech-to-Speech MT Evaluation Slides
- Lena Marg, Rating Evaluation Methods through Correlation Slides
- Federico Gaspari, Antonio Toral, Arle Lommel, Stephen Doherty, Josef van Genabith and Andy Way, Relating Translation Quality Barriers to Source-Text Properties Slides
13:00 – 14:00 – Lunch break
14:00 – 15:00 – Hands-On Session 1
Comprehension Task
15:00 – 16:00 – Hands-On Session 2
Post-Editing Task
16:00 – 16:30 – Coffee break
16:30 – 17:30 – Hands-On Session 3
Annotation Tasks
Task Assignments
17:30 – 18:00 – Discussion, Potential for Future Collaboration, Next Steps, and Conclusion