The 13th Workshop on Innovative Use of NLP for Building Educational Applications

NAACL 2018 Workshops

Pre-Workshop Information - Sponsors - Workshop Description - Shared Task on Second Language Acquisition Modeling - Shared Task on Complex Word Identification - Important Dates - Submission Information - Presentation Information - Workshop Program - Organizing Committee - Program Committee - Related Links

Conference: NAACL 2018

Organization: Joel Tetreault (Grammarly), Jill Burstein (Educational Testing Service), Ekaterina Kochmar (University of Cambridge), Claudia Leacock (Grammarly), Helen Yannakoudakis (University of Cambridge)

Contact Email:

Date : June 05, 2018

Venue : New Orleans, USA

Post-Workshop Information

Pre-Workshop Information

For those attending the BEA13 Workshop in New Orleans on Tuesday, June 05, come back here to this section for quick updates.


We're excited to announce our sponsors for the BEA13 Workshop! Our Gold Sponsors are Duolingo, Grammarly, National Board of Medical Examinders and Turnitin / Lightside Labs. Our Silver Sponsors are Educational Testing Service and iLexIR. Cognii and Newsela are our bronze level sponsors. If you or your company or institution are interested in sponsoring the BEA13, please send us an email at Sponsorship goes toward subsidizing dinner for students attending the workshop and free t-shirts with registration.

Gold Level Sponsors

Silver Level Sponsors

Bronze Level Sponsors

Workshop Description

The BEA Workshop is a leading venue for NLP innovation for educational applications. It is one of the largest one-day workshops in the ACL community. The workshop’s continuous growth illustrates an alignment between societal need and technology advances. NLP capabilities now support an array of learning domain knowledge, including writing, speaking, reading, science, and mathematics, and the related intra- (e.g., self-confidence) and inter-personal (e.g., peer collaboration) domains that support achievement in the learning domains. Within these domains, the community continues to develop and deploy innovative NLP approaches for use in educational settings. In the writing and speech domains, automated writing evaluation (AWE) and speech scoring applications, respectively, are commercially deployed in high-stakes assessment, and instructional contexts (including Massive Open Online Courses (MOOCs), and K-12 settings). Commercially-deployed plagiarism detection in K-12 and higher education settings is also prevalent. The current educational and assessment landscape in K-12, higher education, and adult learning (in academic and workplace settings) fosters a strong interest in technologies that yield user log data that can be leveraged for analytics that support proficiency measures for complex constructs across learning domains. For writing, there is a focus on innovation that supports writing tasks requiring source use, argumentative discourse, and factual content accuracy. For speech, there is an interest in advancing automated scoring to include the evaluation of discourse and content features in responses to spoken assessments. General advances in speech technology have promoted a renewed interest in spoken dialog and multimodal systems for instruction and assessment, for instance, for workplace interviews and simulated teaching environments. The explosive growth of mobile applications for game-based and simulation applications for instruction and assessment is another place where NLP has begun to play a large role, especially in language learning.

NLP for educational applications has gained visibility outside of the NLP community. First, the Hewlett Foundation reached out to public and private sectors and sponsored two competitions: one for automated essay scoring, and the other for scoring of short response items. The motivation driving these competitions was to engage the larger scientific community in this enterprise. Learning @ Scale is a relatively new venue for NLP research in education. MOOCs now incorporate AWE systems to manage several thousand assignments that may be received during a single MOOC course. MOOCs for Refugees have more recently popped up in response to the current social situations. Courses include language learning, and we can imagine that AWE and other NLP capabilities could support coursework. Another breakthrough for educational applications within the CL community is the presence of a number of shared-task competitions over the last four years – including three shared tasks on grammatical error detection and correction alone. NLP/Education shared tasks, typically in the area of grammar-error detection, have seen new areas of research, such as the “Automated Evaluation of Scientific Writing” at BEA11 and Native Language Identification at BEA12. All of these competitions increased the visibility of, and interest in, our field. In conjunction with the International Joint Conference on Natural Language Processing (ACL-IJCNLP) 2015, the Natural Language Processing Techniques for Educational Applications (NLP-TEA) workshop had a shared task in Chinese error diagnosis, and NLP-TEA had additional shared tasks at the 2016, and a fourth workshop in 2017 co-located with IJCNLP.

The 13th BEA workshop will have oral presentation sessions and a large poster session in order to maximize the amount of original work presented. We expect that the workshop will continue to expose the NLP community to technologies that identify novel opportunities for the use of NLP in education in English, and languages other than English. The workshop will solicit both full papers and short papers for either oral or poster presentation. We will solicit papers that incorporate NLP methods, including, but not limited to: automated scoring of open-ended textual and spoken responses; game-based instruction and assessment; educational data mining; intelligent tutoring; peer review, grammatical error detection; learner cognition; spoken dialog; multimodal applications; tools for teachers and test developers; and use of corpora. Research that incorporates NLP methods for use with mobile and game-based platforms will be of special interest. Specific topics include:

Shared Task on Second Language Acquisition Modeling

Task Description

The workshop will host a Shared Task on Second Language Acquisition Modeling (SLAM), using data provided by Duolingo, a popular free online computer-aided language learning (CALL) platform. Teams will be provided with “traces” of all translation and transcription exercises from 800+ language learners — annotated for errors — spanning their first 100 days of activity on Duolingo. The task is then to predict errors made by a held-out set of 100+ language learners over their first 100 days. There will be four tracks for learners of English, Spanish, French and German. We believe that this task presents several new and interesting dimensions for research in Second Language Acquisition modeling: (1) subjects are mostly beginners in their respective L2s, (2) success will likely require teams to model learning — and forgetting — over time, and (3) teams are encouraged to use features which generalize across a variety of languages (hence 4 tracks).


Task Organizers

Burr Settles (Duolingo), Erin Gustafson (Duolingo), Masato Hagiwara (Duolingo), Bozena Pajak (Duolingo), Joseph Rollinson (Duolingo), Chris Brust (Duolingo), Hideki Shima (Duolingo), Nitin Madnani (Educational Testing Service)

Shared Task on Complex Word Identification

Task Description

Over the past decade a number of studies have been published on automatic text simplification (Specia, 2010; Saggion et al. 2015; Štajner, 2015). Text simplification systems aim to facilitate reading comprehension to different target readerships such as foreign language learners, and native speakers with low literacy levels or various kinds of reading impairments. Two main factors that impact reading comprehension addressed by these systems are lexical complexity and syntactic complexity.

Many lexical simplification systems have been proposed up to this date (Glavaš and Štajner, 2015; Paetzold and Specia, 2016). It has been shown that those systems which have a separate complex word identification (CWI) module at the beginning of their pipeline outperform those systems which treat all words as potentially complex (Paetzold and Specia, 2015). Therefore, automatic identification of words that are difficult for a given target population is an important step for building better performing lexical simplification systems. This step is known as complex word identification (CWI) (Shardlow, 2013).

The first shared task on CWI was organized at the SemEval 2016 (Paetzold and Specia, 2016b). It featured 21 teams that competed submitting 42 systems trained to predict whether words in a given context were complex or non-complex for a non-native English speaker. Following the success of the first CWI shared task at SemEval 2016 we propose the organization of a second edition of the challenge at the BEA workshop 2018.

The first edition of the CWI challenge included only English data aimed at non-native English speakers, whereas the second edition will feature a multilingual dataset (Yimam, 2017a, 2017b) and four individual tracks: (1) English monolingual CWI, (2) Spanish monolingual CWI, (3) German monolingual CWI, (4) Multilingual CWI with a French test set.


Task Organizers

Chris Biemann (University of Hamburg), Shervin Malmasi (Harvard Medical School), Gustavo Paetzold (University of Sheffield), Lucia Specia (University of Sheffield), Sanja Štajner (University of Mannheim), Anais Tack (KU Leuven), Seid Muhie Yimam (University of Hamburg), Marcos Zampieri (University of Wolverhampton)

Important Dates

Submission Information

We will be using the NAACL submission guidelines and style files for the BEA13 Workshop this year. Authors are invited to submit a full paper of up to 8 pages of content with unlimited pages for references. We also invite short papers of up to 4 pages of content, including unlimited pages for references. Supplementary files are welcome (for both long and short papers), as long as they follow the NAACL 2018 Guidelines. Final camera ready versions of accepted papers will be given an additional page of content to address reviewer comments.

Previously published papers cannot be accepted. The submissions will be reviewed by the program committee. As reviewing will be blind, please ensure that papers are anonymous. Self-references that reveal the author's identity, e.g., "We previously showed (Smith, 1991) ...", should be avoided. Instead, use citations such as "Smith previously showed (Smith, 1991) ...".

We have also included conflict of interest in the submission form. You should mark all potential reviewers who have been authors on the paper, are from the same research group or institution, or who have seen versions of this paper or discussed it with you.

We will be using the START conference system to manage submissions:

Presentation Information

Oral Presentations: Long papers accepted for oral presentations are allotted 20 minutes for the talk and 5 minutes for questions. Shared Task Description papers and short papers that are accepted for oral presentations are allotted 15 minutes for the talk and 5 minutes for questions.

Poster Presentations: All papers accepted for a poster presentation will be presented in the session after lunch between 2:00 and 3:30. The posterboards will be self-standing, on top of tables (giving room for laptops, business cards, handouts, etc). Poster boards are 4 by 8 feet (121 x 242 cm) in size, and they can accommodate 2’ x 3’ or A0 sized posters in both landscape and portrait formats, or 4’x6’ in landscape format.

The poster session is split in two 45 minute segments. We have split the BEA, CWI Shared Task and SLAM Shared Task papers across the two segments.

Workshop Program

[ BEA13 Proceedings ]
8:30 - 9:00 Loading in of Oral Presentations
9:00 - 9:15 Opening Remarks
[ pdf ] [ slides ]
9:15 - 9:40 Using exemplar responses for training and evaluating automated speech scoring systems
Anastassia Loukina, Klaus Zechner, James Bruno and Beata Beigman Klebanov
[ pdf ] [ slides ]
9:40 - 10:05 Using Paraphrasing and Memory-Augmented Models to Combat Data Sparsity in Question Interpretation with a Virtual Patient Dialogue System
Lifeng Jin, David King, Amad Hussein, Michael White and Douglas Danforth
[ pdf ] [ slides ]
10:05 - 10:30 Predicting misreadings from gaze in children with reading difficulties
Joachim Bingel, Maria Barrett and Sigrid Klerke
[ pdf ] [ slides ]
10:30 - 11:00 Break
11:00 - 11:25 Automatic Input Enrichment for Selecting Reading Material: An Online Study with English Teachers
Maria Chinkina, Ankita Oswal and Detmar Meurers
[ pdf ] [ slides ]
11:25 - 11:50 Estimating Linguistic Complexity for Science Texts
Farah Nadeem and Mari Ostendorf
[ pdf ] [ slides ]
11:50 - 12:10 Second Language Acquisition Modeling
Burr Settles, Chris Brust, Erin Gustafson, Masato Hagiwara and Nitin Madnani
[ pdf ] [ slides ]
12:10 - 12:30 A Report on the Complex Word Identification Shared Task 2018
Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo Paetzold, Lucia Specia, Sanja Štajner, Anaïs Tack and Marcos Zampieri
[ pdf ] [ slides ]
12:30 - 14:00 Lunch
14:00 - 15:30 BEA13 Poster and Shared Task Session
14:00 - 14:45 Poster Session A
BEA Papers
Towards Single Word Lexical Complexity Prediction
David Alfter and Elena Volodina
[ pdf ]
COAST - Customizable Online Syllable Enhancement in Texts. A flexible framework for automatically enhancing reading materials
Heiko Holz, Zarah Weiss, Oliver Brehm and Detmar Meurers
[ pdf ]
Annotating picture description task responses for content analysis
Levi King and Markus Dickinson
[ pdf ] [ poster ]
Annotating Student Talk in Text-based Classroom Discussions
Luca Lugini, Diane Litman, Amanda Godley and Christopher Olshefski
[ pdf ]
Toward Automatically Measuring Learner Ability from Human-Machine Dialog Interactions using Novel Psychometric Models
Vikram Ramanarayanan and Michelle LaMar
[ pdf ]
Generating Feedback for English Foreign Language Exercises
Björn Rudzewitz, Ramon Ziai, Kordula De Kuthy, Verena Möller, Florian Nuxoll and Detmar Meurers
[ pdf ]
NT2Lex: A CEFR-Graded Lexical Resource for Dutch as a Foreign Language Linked to Open Dutch WordNet
Anaïs Tack, Thomas François, Piet Desmet and Cédrick Fairon
[ pdf ] [ poster ]
Experiments with Universal CEFR Classification
Sowmya Vajjala and Taraka Rama
[ pdf ]
Chengyu Cloze Test
Zhiying Jiang, Boliang Zhang, Lifu Huang and Heng Ji
[ pdf ] [ poster ]
CWI Shared Task Papers
LaSTUS/TALN at Complex Word Identification (CWI) 2018 Shared Task
Ahmed AbuRa’ed and Horacio Saggion
[ pdf ]
Cross-lingual complex word identification with multitask learning
Joachim Bingel and Johannes Bjerva
[ pdf ] [ poster ]
UnibucKernel: A kernel-based learning method for complex word identification
Andrei Butnaru and Radu Tudor Ionescu
[ pdf ] [ poster ]
CAMB at CWI Shared Task 2018: Complex Word Identification with Ensemble- Based Voting
Sian Gooding and Ekaterina Kochmar
[ pdf ]
Complex Word Identification Based on Frequency in a Learner Corpus
Tomoyuki Kajiwara and Mamoru Komachi
[ pdf ] [ poster ]
The Whole is Greater than the Sum of its Parts: Towards the Effectiveness of Voting Ensemble Classifiers for Complex Word Identification
Nikhil Wani, Sandeep Mathias, Jayashree Aanand Gajjam and Pushpak Bhattacharyya
[ pdf ]
SLAM Shared Task Papers
Grotoco@SLAM: Second Language Acquisition Modeling with Simple Features, Learners and Task-wise Models
Sigrid Klerke, Héctor Martínez Alonso and Barbara Plank
[ pdf ]
Context Based Approach for Second Language Acquisition
Nihal V. Nayak and Arjun R. Rao
[ pdf ]
Second Language Acquisition Modeling: An Ensemble Approach
Anton Osika, Susanna Nilsson, Andrii Sydorchuk, Faruk Sahin and Anders Huss
[ pdf ] [ poster ]
Modeling Second-Language Learning from a Psychological Perspective
Alexander Rich, Pamela Osborn Popp, David Halpern, Anselm Rothe and Todd Gureckis
[ pdf ] [ poster ]
A Memory-Sensitive Classification Model of Errors in Early Second Language Learning
Brendan Tomoschuk and Jarrett Lovelett
[ pdf ] [ poster ]
14:45 - 15:30 Poster Session B
BEA Papers
Annotation and Classification of Sentence-level Revision Improvement
Tazin Afrin and Diane Litman
[ pdf ]
Language Model Based Grammatical Error Correction without Annotated Training Data
Christopher Bryant and Ted Briscoe
[ pdf ] [ poster ]
A Semantic Role-based Approach to Open-Domain Automatic Question Generation
Michael Flor and Brian Riordan
[ pdf ]
Automated Content Analysis: A Case Study of Computer Science Student Summaries
Yanjun Gao, Patricia M.Davies and Rebecca J. Passonneau
[ pdf ]
Toward Data-Driven Tutorial Question Answering with Deep Learning Conversational Models
Mayank Kulkarni and Kristy Boyer
[ pdf ]
Distractor Generation for Multiple Choice Questions Using Learning to Rank
Chen Liang, Xiao Yang, Neisarg Dave, Drew Wham, Bart Pursel and C Lee Giles
[ pdf ]
A Portuguese Native Language Identification Dataset
Iria del Río Gayo, Marcos Zampieri and Shervin Malmasi
[ pdf ]
OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification
Sowmya Vajjala and Ivana Lucic
[ pdf ]
The Effect of Adding Authorship Knowledge in Automated Text Scoring
Meng Zhang, Xie Chen, Ronan Cummins, Øistein E. Andersen and Ted Briscoe
[ pdf ]
CWI Shared Task Papers
SB@GU at the Complex Word Identification 2018 Shared Task
David Alfter and Ildikó Pilán
[ pdf ]
Complex Word Identification: Convolutional Neural Network vs. Feature Engineering
Segun Taofeek Aroyehun, Jason Angel, Daniel Alejandro Pérez Alvarez and Alexander Gelbukh
[ pdf ] [ poster ]
Deep Learning Architecture for Complex Word Identification
Dirk De Hertog and Anaïs Tack
[ pdf ] [ poster ]
NILC at CWI 2018: Exploring Feature Engineering and Feature Learning
Nathan Hartmann and Leandro Borges dos Santos
[ pdf ] [ poster ]
Complex Word Identification Using Character n-grams
Maja Popović
[ pdf ]
SLAM Shared Task Papers
Predicting Second Language Learner Successes and Mistakes by Means of Conjunctive Features
Yves Bestgen
[ pdf ]
Feature Engineering for Second Language Acquisition Modeling
Guanliang Chen, Claudia Hauff and Geert-Jan Houben
[ pdf ] [ poster ]
TMU System for SLAM-2018
Masahiro Kaneko, Tomoyuki Kajiwara and Mamoru Komachi
[ pdf ] [ poster ]
Deep Factorization Machines for Knowledge Tracing
Jill-Jênn Vie
[ pdf ] [ poster ]
CLUF: a Neural Model for Second Language Acquisition Modeling
Shuyao Xu, Jin Chen and Long Qin
[ pdf ] [ poster ]
Neural sequence modelling for learner error prediction
Zheng Yuan
[ pdf ]
15:30 - 16:00 Break
16:00 - 16:25 Automatic Distractor Suggestion for Multiple-Choice Tests Using Concept Embeddings and Information Retrieval
Le An Ha and Victoria Yaneva
[ pdf ]
16:25 - 16:50 Co-Attention Based Neural Network for Source-Dependent Essay Scoring
Haoran Zhang and Diane Litman
[ pdf ] [ slides ]
16:50 - 17:15 Cross-Lingual Content Scoring
Andrea Horbach, Sebastian Stennmanns and Torsten Zesch
[ pdf ] [ slides ]
17:15 - 17:30 Closing Remarks
[ slides ]
6:00 - Post-workshop dinner (to be determined)

Organizing Committee

Program Committee

Related Links