The 12th Workshop on Innovative Use of NLP for Building Educational Applications

EMNLP 2017 Workshops

Sponsors - Pre-Workshop Information - Workshop Description - NLI Shared Task - Submission Info - Important Dates - Presentation Information - Workshop Program - Organizing Committee - Program Committee - Related Links - Upcoming Education / NLP Events

Conference: EMNLP 2017

Organization: Joel Tetreault (Grammarly), Jill Burstein (Educational Testing Service), Ekaterina Kochmar (University of Cambridge), Claudia Leacock (Consultant), Helen Yannakoudakis (University of Cambridge)

Contact Email:

Date : September 08, 2017

Venue : Copenhagen, Denmark

Pre-Workshop Information

For those attending the BEA12 Workshop this Friday, September 08, there are a few notes:


Educational Testing Service, Pacific Metrics, National Board of Medical Examinders and iLexIR. Cognii is our bronze level sponsor. If you or your company or institution are interested in sponsoring the BEA12, please send us an email at Sponsorship goes toward subsidizing dinner for students attending the workshop and free t-shirts with registration.

Gold Level Sponsors

Silver Level Sponsors

Bronze Level Sponsors

Workshop Description

The BEA Workshop, one of the largest one-day workshops in the ACL community, is a leading venue for NLP innovation for educational applications. The workshop’s continuous growth illustrates an alignment between societal need and technological advances. NLP capabilities now support an array of learning domains, including writing, speaking, reading, science, and mathematics, as well as the related intra- (e.g., self-confidence) and inter-personal (e.g., collaboration) domains that support achievement in those domains. Automated writing evaluation (AWE) and speech scoring applications are commercially deployed in high-stakes assessment and instructional contexts, including primary and secondary educational settings. Commercially-deployed plagiarism detection is also prevalent. The current educational and assessment landscape in K-12, higher education, and adult learning (in academic and workplace settings) fosters a strong interest in technologies that yield analytics to support proficiency measures for complex constructs. For writing, there is a focus on innovation that supports writing tasks requiring source use, argumentative discourse, and factual content accuracy. For speech, there is an interest in advancing automated scoring to include the evaluation of discourse and content features in responses to spoken assessments. General advances in speech technology have promoted a renewed interest in spoken dialog and multimodal systems for instruction and assessment in, for example, workplace interviews and simulated teaching environments. The explosive growth of mobile applications for game-based and simulation applications for instruction and assessment is another place where NLP has begun to play a large role, especially in language learning.

NLP for educational applications has gained visibility outside of the NLP community. First, the Hewlett Foundation reached out to public and private sectors and sponsored two competitions – one for automated essay scoring, and another for scoring of short response items – to engage the larger scientific community in this enterprise. Subsequently, EdX, a non-profit enterprise founded by Harvard and MIT, announced the release of software designed to automatically grade text using AI technologies. Learning@Scale, a relatively new venue for NLP research in education, promotes interdisciplinary research on learning and teaching. Massive Open Online Courses (MOOCs) now incorporate AWE systems to manage the thousands of assignments that may be received during a single MOOC course. MOOCs for Refugees have more recently popped up in response to current social situations. Courses include language learning, and AWE and other NLP capabilities could support the coursework.

Research into education-related problems has also continued to grow within the NLP community. The last five years saw eight shared tasks (including five on grammatical error correction), two workshops on NLP and Computer-Assisted Language Learning (TEA series), and two sessions of the ACL devoted exclusively to the role of NLP in assisting language learners. Three of the shared tasks were sponsored by BEA. At ACL 2016, there were over 12 papers devoted to NLP in education, the highest number ever at a conference of this caliber. All of this has served to increase the visibility of, and interest in, the field of education applications in NLP.

The 12th workshop will have oral presentation sessions and a large poster session in order to maximize the amount of original work presented (the workshop normally has 50+ attendees). We will invite both full papers and short papers on topics including: automated scoring of textual and spoken responses, intelligent tutoring, peer review, grammatical error detection/correction, learner cognition, spoken dialog, multimodal applications, tools for teachers and test developers, and use of corpora. Research that incorporates NLP for use with mobile and game-based platforms will be of special interest. Specific topics include:

Finally, we will be adhering to ACL's anti-harrassment policy.

NLI Shared Task 2017

The workshop will also host a Shared Task on Native Language Identification (NLI). NLI is the process of automatically identify the native language (L1) of a non-native speaker based solely on language that he or she produces in another language. Two previous shared tasks on NLI have been organized in which the task was to identify the native language of non-native speakers of English-based on essays and spoken responses they provided during a standardized assessment of academic English proficiency. The first shared task was based on the essays only and was also held with the BEA workshop in 2013. It was a total success with 29 teams competing, making it one of the largest shared tasks that year. Three years later, Computational Paralinguistics Challenge at Interspeech 2016 hosted a sub-challenge on identifying the native language based solely on the spoken responses.

This year's shared task combines the inputs from the two previous tasks. There will be three tracks: NLI on the essay only, NLI on the speech response only, and NLI using both responses from a test taker. We feel this will make for a more challenging shared task while building on the methods and results from the previous two shared tasks. The training and development data for the shared task will be available in February 2017.

The organizing committee for the shared task is: Aoife Cahill (Educational Testing Service), Keelan Evanini (Educational Testing Service), Shervin Malmasi (Harvard Medical School), Joel Tetreault (Grammarly).

More details can be found on the shared task website:

Submission Information

We will be using the EMNLP Submission Guidelines for the BEA12 Workshop this year. (Please note we initially had the ACL Guidelines as the point of reference, but use the EMNLP ones. The two are extremely similar and you shouldn't see any differences when it comes to formatting and paper length). Authors are invited to submit a full paper of up to 8 pages of content with unlimited pages for references. We also invite short papers of up to 4 pages of content, including unlimited pages for references. Final camera ready versions of accepted papers will be given an additional page of content to address reviewer comments.

Papers which describe systems are also invited to give a demo of their system. If you would like to present a demo in addition to presenting the paper, please make sure to select either "full paper + demo" or "short paper + demo" under "Submission Category" in the START submission page.

Previously published papers cannot be accepted. The submissions will be reviewed by the program committee. As reviewing will be blind, please ensure that papers are anonymous. Self-references that reveal the author's identity, e.g., "We previously showed (Smith, 1991) ...", should be avoided. Instead, use citations such as "Smith previously showed (Smith, 1991) ...".

We have also included conflict of interest in the submission form. You should mark all potential reviewers who have been authors on the paper, are from the same research group or institution, or who have seen versions of this paper or discussed it with you.

We will be using the START conference system to manage submissions:

As in the above, please make sure you are using the EMNLP Submission Guidelines. Thanks!

Important Dates

Presentation Information

Oral Presentations: Long papers accepted for oral presentations are allotted 20 minutes for the talk and 5 minutes for questions. Short papers that are accepted for oral presentations are allotted 15 minutes for the talk and 5 minutes for questions.

Poster Presentations: All papers accepted for a poster presentation will be presented in the session after lunch between 2:00 and 3:30. The posterboards will be self-standing, on top of tables (giving room for laptops, business cards, handouts, etc). The posters should be sized for A0 Landscape. Double-sided tape, pushpins, etc. for affixing the posters to the boards will be provided.

Workshop Program

[ BEA12 Proceedings ]
8:45 - 9:00 Loading in of Oral Presentations
9:00 - 9:15 Opening Remarks
9:15 - 9:40 Question Difficulty – How to Estimate Without Norming, How to Use for Automated Grading
Ulrike Pado
9:40 - 10:05 Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System
Lifeng Jin, Michael White, Evan Jaffe, Laura Zimmerman and Douglas Danforth
10:05 - 10:30 Continuous fluency tracking and the challenges of varying text complexity
Beata Beigman Klebanov, Anastassia Loukina, John Sabatini and Tenaha O’Reilly
10:30 - 11:00 Break
11:00 - 11:25 Auxiliary Objectives for Neural Error Detection Models
Marek Rei and Helen Yannakoudakis
11:25 - 11:50 Linked Data for Language-Learning Applications
Robyn Loughnane, Kate McCurdy, Peter Kolb and Stefan Selent
11:50 - 12:10 Predicting Specificity in Classroom Discussion
Luca Lugini and Diane Litman
12:10 - 12:35 A Report on the 2017 Native Language Identification Shared Task
Shervin Malmasi, Keelan Evanini, Aoife Cahill, Joel Tetreault, Robert Pugh, Christopher Hamill, Diane Napolitano and Yao Qian
12:35 - 14:00 Lunch (Øksnehallen)
14:00 - 15:30 BEA12 Poster and Demo Session
14:00 - 14:45 BEA12 Poster and Demo Session A
Evaluation of Automatically Generated Pronoun Reference Questions
Arief Yudha Satria and Takenobu Tokunaga
Predicting Audience’s Laughter During Presentations Using Convolutional Neural Network
Lei Chen and Chong Min Lee
Collecting fluency corrections for spoken learner English
Andrew Caines, Emma Flint and Paula Buttery
Exploring Relationships Between Writing & Broader Outcomes With Automated Writing Evaluation
Jill Burstein, Dan McCaffrey, Beata Beigman Klebanov and Guangming Ling
An Investigation into the Pedagogical Features of Documents
Emily Sheng, Prem Natarajan, Jonathan Gordon and Gully Burns
Combining Multiple Corpora for Readability Assessment for People with Cognitive Disabilities
Victoria Yaneva, Constantin Orasan, Richard Evans and Omid Rohanian
Automatic Extraction of High-Quality Example Sentences for Word Learning Using a Determinantal Point Process
Arseny Tolmachev and Sadao Kurohashi
Distractor Generation for Chinese Fill-in-the-blank Items
Shu Jiang and John Lee
An Error-Oriented Approach to Word Embedding Pre-Training
Youmna Farag, Marek Rei and Ted Briscoe
Investigating neural architectures for short answer scoring
Brian Riordan, Andrea Horbach, Aoife Cahill, Torsten Zesch and Chong Min Lee
Human and Automated CEFR-based Grading of Short Answers
Anaïs Tack, Thomas François, Sophie Roekhaut and Cédrick Fairon
GEC into the future: Where are we going and how do we get there?
Keisuke Sakaguchi, Courtney Napoles and Joel Tetreault
Detecting Off-topic Responses to Visual Prompts
Marek Rei
Combining Textual and Speech Features in the NLI Task Using State-of-the-Art Machine Learning Techniques
Pavel Ircing, Jan Svec, Zbynek Zajic, Barbora Hladka and Martin Holub
Native Language Identification Using a Mixture of Character and Word N-grams
Elham Mohammadi, Hadi Veisi and Hessam Amini
Ensemble Methods for Native Language Identification
Sophia Chan, Maryam Honari Jahromi, Benjamin Benetti, Aazim Lakhani and Alona Fyshe
Can string kernels pass the test of time in Native Language Identification?
Radu Tudor Ionescu and Marius Popescu
Neural Networks and Spelling Features for Native Language Identification
Johannes Bjerva, Gintare Grigonyte, Robert Östling and Barbara Plank
A study of N-gram and Embedding Representations for Native Language Identification
Sowmya Vajjala and Sagnik Banerjee
A Shallow Neural Network for Native Language Identification with Character Ngrams
Yunita Sari, Muhammad Rifqi Fatchurrahman and Meisyarah Dwiastuti
Fewer features perform well at Native Language Identification task
Taraka Rama and Ça˘grı Çöltekin
14:45 - 15:30 BEA11 Poster and Demo Session B
Structured Generation of Technical Reading Lists
Jonathan Gordon, Stephen Aguilar, Emily Sheng and Gully Burns
Effects of Lexical Properties on Viewing Time per Word in Autistic and Neurotypical Readers
Sanja Štajner, Victoria Yaneva, Ruslan Mitkov and Simone Paolo Ponzetto
Transparent text quality assessment with convolutional neural networks
Robert Östling and Gintare Grigonyte
Artificial Error Generation with Machine Translation and Syntactic Patterns
Marek Rei, Mariano Felice, Zheng Yuan and Ted Briscoe
Modelling semantic acquisition in second language learning
Ekaterina Kochmar and Ekaterina Shutova
Multiple Choice Question Generation Utilizing An Ontology
Katherine Stasaski and Marti A. Hearst
Simplifying metaphorical language for young readers: A corpus study on news text
Magdalena Wolska and Yulia Clausen
Language Based Mapping of Science Assessment Items to Skills
Farah Nadeem and Mari Ostendorf
Connecting the Dots: Towards Human-Level Grammatical Error Correction
Shamil Chollampatt and Hwee Tou Ng
Question Generation for Language Learning: From ensuring texts are read to supporting learning
Maria Chinkina and Detmar Meurers
Systematically Adapting Machine Translation for Grammatical Error Correction
Courtney Napoles and Chris Callison-Burch
Fine-grained essay scoring of a complex writing task for native speakers
Andrea Horbach, Dirk Scholten-Akoun, Yuning Ding and Torsten Zesch
Exploring Optimal Voting in Native Language Identification
Cyril Goutte and Serge Léger
CIC-FBK Approach to Native Language Identification
Ilia Markov, Lingzhen Chen, Carlo Strapparava and Grigori Sidorov
The Power of Character N-grams in Native Language Identification
Artur Kulmizev, Bo Blankers, Johannes Bjerva, Malvina Nissim, Gertjan van Noord, Barbara Plank and Martijn Wieling
Classifier Stacking for Native Language Identification
Wen Li and Liang Zou
Native Language Identification on Text and Speech
Marcos Zampieri, Alina Maria Ciobanu and Liviu P. Dinu
Native Language Identification using Phonetic Algorithms
Charese Smiley and Sandra Kübler
A deep-learning based native-language classification by using a latent semantic analysis for the NLI Shared Task 2017
Yoo Rhee Oh, Hyung-Bae Jeon, Hwa Jeon Song, Yun-Kyung Lee, Jeon-Gue Park and Yun-Keun Lee
Fusion of Simple Models for Native Language Identification
Fabio Kepler, Ramón Astudillo and Alberto Abad
Stacked Sentence-Document Classifier Approach for Improving Native Language Identification
Andrea Cimino and Felice Dell’Orletta
15:30 - 16:00 Break
16:00 - 16:25 Using Gaze to Predict Text Readability
Ana Valeria Gonzalez-Garduño and Anders Søgaard
16:25 - 16:50 Annotating Orthographic Target Hypotheses in a German L1 Learner Corpus
Ronja Laarmann-Quante, Katrin Ortmann, Anna Ehlert, Maurice Vogel and Stefanie Dipper
16:50 - 17:15 A Large Scale Quantitative Exploration of Modeling Strategies for Content Scoring
Nitin Madnani, Anastassia Loukina and Aoife Cahill
17:15 - 17:30 Closing Remarks
6:00 - Post-workshop dinner
Al Diwan
[ Directions to Vesterbrogade 94, 1620 København V, Denmark ]

Organizing Committee

Program Committee

Related Links

Other 2016 and 2017 Educational/NLP Events