The 10th Workshop on Innovative Use of NLP for Building Educational Applications

NAACL 2015 Workshops

Sponsors - Pre-Workshop Information - Post-Workshop Information - Workshop Description - Submission Info - Important Dates - Presentation Info - Workshop Program - Program Committee - Related Links - Upcoming Education / NLP Events

Conference: NAACL 2015

Organization: Joel Tetreault (Yahoo Labs), Jill Burstein (Educational Testing Service), Claudia Leacock (McGraw-Hill Education, CTB)

Contact Email:

Date : June 04, 2015

Venue : Denver, CO, USA


We are pleased to announce that Appen, McGraw-Hill Education/CTB, Educational Testing Service, Grammarly, Turnitin Lightside Labs, Pacific Metrics and Pearson are all gold level sponsors of the BEA10 workshop and American Institutes for Research is a silver level sponsor! If you or your company or institution are interested in sponsoring the BEA10, please send us an email at Sponsorship goes toward subsidizing dinner for students attending the workshop and free t-shirts with registration.

Gold Level Sponsors

Silver Level Sponsors

Pre-Workshop Information

For those attending the BEA10 Workshop this Thursday, June 04, there are a few notes:

Post-Workshop Information

The tenth anniversary of the workshop was a success! At the workshop we had one of our largest attendances ever (certainly the largest without a shared task), a great crop of interesting and diverse papers, as well as great t-shirts and donuts courtesy of our sponsors above. :) Some final information notes:

Workshop Description

We are excited to be proposing a 10th anniversary BEA workshop. Since starting in 1997, the BEA workshop, now one of the largest workshops at NAACL/ACL, has become one of the leading venues for publishing innovative work which uses NLP to develop educational applications.

The consistent interest and growth of the workshop has clear ties to societal need and related advances in the technology, and the maturity of the NLP/education field. NLP capabilities now support an array of learning domains, including writing, speaking, reading, and mathematics. Within these domains, the community continues to develop and deploy innovative NLP approaches for use in educational settings. In the writing and speech domains, automated writing evaluation (AWE) and speech scoring applications, respectively, are commercially deployed in high-stakes assessment and instructional settings, including Massive Open Online Courses (MOOCs). We also see widely-used commercial applications for plagiarism detection and peer review. Major advances in speech technology, have made it possible to include speech in both assessment and Intelligent Tutoring Systems (ITS). There has been a renewed interest in spoken dialog and multi-modal systems for instruction and assessment. We are also seeing explosive growth of mobile applications for game-based applications for instruction and assessment. The current educational and assessment landscape, especially in the United States, continues to foster a strong interest and high demand that pushes the state-of-the-art in AWE capabilities to expand the analysis of written responses to writing genres other than those traditionally found in standardized assessments, especially writing tasks requiring use of sources and argumentative discourse.

The use of NLP in educational applications has gained visibility outside of the NLP community. First, the Hewlett Foundation reached out to public and private sectors and sponsored two competitions: one for automated essay scoring , and the other for scoring of short answer, fact-based response items . The motivation driving these competitions was to engage the larger scientific community in this enterprise. MOOCs are now beginning to incorporate AWE systems to manage the thousands of constructed-response assignments collected during a single MOOC course. Learning@Scale is a new venue for discussing NLP research in education. Another breakthrough for educational applications within the CL community is the presence of a number of shared-task competitions over the last three years. There have been three shared tasks on grammatical error correction with the most recent edition hosted at CoNLL 2014. In 2014 alone, there were four shared tasks for NLP and Education-related areas.

In 2015, we expect that the workshop (consistent with the nine previous workshops at ACL and NAACL/HLT), will continue to expose the NLP research community to technologies that identify novel opportunities for the use of NLP techniques and tools in educational applications. This BEA10 workshop will solicit both full papers and short papers for oral and poster presentations. We will solicit papers for educational applications that incorporate NLP methods, including, but not limited to: automated scoring of open-ended textual and spoken responses; game-based instruction and assessment; intelligent tutoring; peer review, grammatical error detection; learner cognition; spoken dialog; multi-modal applications; tools for teachers and test developers; and use of corpora. Research that incorporates NLP methods for use with mobile and game-based platforms, and academic ePortfolio systems or MOOCs continues to be of special interest. Finally, as this is the 10th anniversary, we invite papers which provide a retrospective view, reflecting on past and current trends in the field, and vision papers which illustrate research directions for growth in the field. Specific topics include:

Submission Information

We will be using the NAACL 2015 Submission Guidelines for the BEA10 Workshop this year. Authors are invited to submit a full paper of up to 9 pages of content with up to 2 additional pages for references. We also invite short papers of up to 5 pages of content, including 2 additional pages for references. Please note that unlike previous years, final, camera ready versions of accepted papers will not be given an additional page to address reviewer comments.

Papers which describe systems are also invited to give a demo of their system. If you would like to present a demo in addition to presenting the paper, please make sure to select either "full paper + demo" or "short paper + demo" under "Submission Category" in the START submission page.

Previously published papers cannot be accepted. The submissions will be reviewed by the program committee. As reviewing will be blind, please ensure that papers are anonymous. Self-references that reveal the author's identity, e.g., "We previously showed (Smith, 1991) ...", should be avoided. Instead, use citations such as "Smith previously showed (Smith, 1991) ...".

Please use the 2015 NAACL style sheets for composing your paper: .

We will be using the START conference system to manage submissions:

Important Dates

Presentation Information

Oral Presentations: Long papers accepted for oral presentations are allotted 20 minutes for the talk and 5 minutes for questions. Short papers that are accepted for oral presentations are allotted 15 minutes for the talk and 5 minutes for questions.

Poster Presentations: All papers accepted for a poster presentation will be presented in the session after lunch between 2:00 and 3:30. The posterboards will be self-standing, on top of tables (giving room for laptops, business cards, handouts, etc). The posterboards measure 36 inches high and 48 inches wide . Double-sided tape, pushpins, etc. for affixing the posters to the boards will be provided.

Workshop Program

[ BEA10 Proceedings ]
8:45 - 9:00 Loading in of Oral Presentations
9:00 - 9:15 Opening Remarks
[ paper ] [ slides ]
9:15 - 9:40 Candidate evaluation strategies for improved difficulty prediction of language tests
Lisa Beinborn, Torsten Zesch and Iryna Gurevych
[ paper ] [ slides ]
9:40 - 10:05 Feature selection for automated speech scoring
Anastassia Loukina, Klaus Zechner, Lei Chen and Michael Heilman
[ paper ] [ slides ]
10:05 - 10:30 Incorporating Coherence of Topics as a Criterion in Automatic Response-to-Text Assessment of the Organization of Writing
Zahra Rahimi, Diane Litman, Elaine Wang and Richard Correnti
[ paper ] [ slides ]
10:30 - 11:00 Break
11:00 - 11:25 Automatic morphological analysis of learner Hungarian
Scott Ledbetter and Markus Dickinson
[ paper ] [ slides ]
11:25 - 11:45 Automated Scoring of Picture-based Story Narration
Swapna Somasundaran, Chong Min Lee, Martin Chodorow and Xinhao Wang
[ paper ] [ slides ]
11:45 - 12:05 Measuring Feature Diversity in Native Language Identification
Shervin Malmasi and Aoife Cahill
[ paper ] [ slides ]
12:05 - 12:25 Automated Evaluation of Scientific Writing: AESW Shared Task Proposal
Vidas Daudaravicius
[ paper ] [ slides ]
12:25 - 2:00 Lunch
2:00 - 3:30 BEA10 Poster and Demo Session
2:00 - 2:45 BEA10 Poster and Demo Session A
Automated Evaluation of Scientific Writing: AESW Shared Task Proposal
Vidas Daudaravicius
[ paper ]
Towards Automatic Description of Knowledge Components
Cyril Goutte, Guillaume Durand and Serge Leger
[ paper ]
Lark Trills for Language Drills: Text-to-speech technology for language learners (Demo)
Elena Volodina and Dijana Pijetlovic
[ paper ]
Reducing Annotation Efforts in Supervised Short Answer Scoring
Torsten Zesch, Michael Heilman and Aoife Cahill
[ paper ]
The Impact of Training Data on Automated Short Answer Scoring Performance
Michael Heilman and Nitin Madnani
[ paper ]
Interpreting Questions with a Log-Linear Ranking Model in a Virtual Patient Dialogue System
Evan Jaffe, Michael White, William Schuler, Eric Fosler-Lussier, Alex Rosenfeld and Douglas Danforth
[ paper ]
The Jinan Chinese Learner Corpus
Maolin Wang, Shervin Malmasi and Mingxuan Huang
[ paper ]
Annotation and Classification of Argumentative Writing Revisions
Fan Zhang and Diane Litman
[ paper ]
Identifying Patterns For Short Answer Scoring Using Graph-based Lexico-Semantic Text Matching
Lakshmi Ramachandran, Jian Cheng and Peter Foltz
[ paper ]
Scoring Persuasive Essays Using Opinions and their Targets
Noura Farra, Swapna Somasundaran and Jill Burstein
[ paper ]
2:45 - 3:30 BEA10 Poster and Demo Session B
Task-Independent Features for Automated Essay Grading
Torsten Zesch, Michael Wojatzki and Dirk Scholten-Akoun
[ paper ]
Oracle and Human Baselines for Native Language Identification
Shervin Malmasi, Joel Tetreault and Mark Dras
[ paper ]
Generating Reference Texts for Short Answer Scoring Using Graph-based Summarization
Lakshmi Ramachandran and Peter Foltz
[ paper ]
Towards Creating Pedagogic Views from Encyclopedic Resources
Ditty Mathew, Dhivya Eswaran and Sutanu Chakraborti
[ paper ]
Judging the Quality of Automatically Generated Gap-fill Question using Active Learning
Nobal Niraula and Vasile Rus
[ paper ]
Evaluating the performance of Automated Text Scoring systems
Helen Yannakoudakis and Ronan Cummins
[ paper ]
Embarrassed or Awkward? Ranking Emotion Synonyms for ESL Learners' Appropriate Wording
Wei-Fan Chen, MeiHua Chen and Lun-Wei Ku
[ paper ]
Preliminary Experiments on Crowdsourced Evaluation of Feedback Granularity
Nitin Madnani, Martin Chodorow, Aoife Cahill, Melissa Lopez, Yoko Futagi and Yigal Attali
[ paper ]
Using PEGWriting® to Support the Writing Motivation and Writing Quality of Eighth-Grade Students: A Quasi-Experimental Study
Trish Martin and Joshua Wilson
[ paper ]
RevUP: Automatic Gap-Fill Question Generation from Educational Texts
Girish Kumar, Rafael Banchs and Luis Fernando D'Haro
[ paper ]
3:30 - 4:00 Break
4:00 - 4:25 Using Learner Data to Improve Error Correction in Adjective–Noun Combinations
Ekaterina Kochmar and Ted Briscoe
[ paper ] [ slides ]
4:25 - 4:50 Using NLP to support scalable assessment of short free text responses
Alistair Willis
[ paper ] [ slides ]
4:50 - 5:15 Automatically Scoring Freshman Writing: A Preliminary Investigation
Courtney Napoles and Chris Callison-Burch
[ paper ] [ slides ]
5:15 - 5:30 Closing Remarks
7:00 - Post-workshop dinner
Rio Grande Mexican Restaurant at 1525 Blake Street
[ directions from hotel to restaurant ]

Program Committee

Related Links

Other Upcoming Educational/NLP Events