The 9th Workshop on Innovative Use of NLP for Building Educational Applications

ACL 2014 Workshops

Sponsors - Pre-Workshop Information - Workshop Description - Invited Speaker - Submission Info - Important Dates - Presentation Info - Workshop Program - Program Committee - Related Links - Upcoming Education / NLP Events

Conference: ACL 2014

Organization: Joel Tetreault (Yahoo! Labs), Jill Burstein (Educational Testing Service), Claudia Leacock (CTB McGraw-Hill)

Contact Email:

Date : June 26, 2014

Venue : Baltimore, MD, USA


BEA9 information


We are pleased to announce that CTB McGraw-Hill, edX, Educational Testing Service, Lightside and Pearson are all gold level sponsors of the BEA9 workshop and American Institutes for Research is a silver level sponsor! If you or your company or institution are interested in sponsoring the BEA9, please send us an email at Sponsorship goes toward subsidizing dinner for students attending the workshop, free t-shirts with registration and the invited speaker (Dr. Norbert Elliot).

Gold Level Sponsors

Silver Level Sponsors

Pre-Workshop Information

For those attending the BEA9 Workshop this Thursday, June 26, there are a few notes:

Workshop Description

The field of NLP and education has dramatically matured since the first BEA workshop in 1997 , where the primary focus was on grammatical error detection. As a community, we have continued to improve existing capabilities and to identify and develop innovative and creative NLP approaches for use in educational settings. In the writing domain, automated writing evaluation systems are now commercially viable, and are used to score millions of test-taker essays on high-stakes assessments. In speech, major advances in speech technology, have made it possible to include speech in both assessment and Intelligent Tutoring Systems (ITS). Spoken constructed responses are now being used in low-stakes and practice applications. Consistent with this, there is also a renewed interest in spoken dialog for instruction and assessment. Relative to continued innovation, the explosive growth of mobile applications has increased interest in game-based applications for instruction and assessment. The current educational and assessment landscape, especially in the United States, continues to foster a strong interest and high demand that pushes the state-of-the-art in automated writing evaluation capabilities to expand the analysis of written responses to writing genres other than those presently found in standardized assessments. Much of the current demand for creative, new educational applications stems from the development of the Common Core State Standards Initiative (CCSSI). CCSSI describes what K-12 students should be learning with regard to reading, writing, speaking, listening, language, and media and technology. The goal of CCSSI is to ensure college- and workplace-readiness across those domains.

In the past few years, the use of NLP in educational applications has gained visibility outside of the computational linguistics (CL) community. First, the Hewlett Foundation reached out to public and private sectors and sponsored two competitions (both inspired by the CCSSI): one for automated essay scoring, and the other for scoring of short response items. The motivation driving these competitions was to engage the larger scientific community in this enterprise. Massive Open Online Courses (MOOCs) are now also beginning to incorporate automated writing evaluation systems to manage the thousands of assignments that may be received during a single MOOC course (New York Times). Another breakthrough for educational applications within the CL community is the presence of a number of shared-task competitions over the last three years. There have been three shared tasks on grammatical error correction with the most recent edition hosted at CoNLL 2013. Also in 2013 there was a SemEval Shared Task on Student Response Analysis and one on Native Language Identification (hosted at the 2013 edition of this workshop). All of these competitions increased the visibility of the research space for NLP for building educational applications. While attendance has continued to be strong for several years, 2013 was a banner year for the BEA workshop as it was the largest ever and had the largest attendance count of any one-day workshop at NAACL.

The 2014 workshop will solicit both full papers and short papers for either oral or poster presentation. Given the broad scope of the workshop, we organize the workshop around three central themes in the educational infrastructure: (1) development of curriculum and assessments; (2) delivery of curriculum and assessments; and (3) reporting of assessment outcomes. In the 2014 workshop, we will solicit papers for educational applications that incorporate NLP methods, including, but not limited to: automated scoring of open-ended textual and spoken responses; game-based instruction and assessment; intelligent tutoring; grammatical error detection; learner cognition; spoken dialog; tools for teachers and test developers; and use of corpora. Research that incorporates NLP methods for use with mobile and game-based platforms, and academic ePortfolio sytems or MOOCs would be of special interest. Since the first workshop in 1997, the BEA workshop series has continued to bring together many NLP subfields, and to foster interaction and collaboration among researchers in academia and industry. The workshop offers a unique venue for researchers to present and discuss their work. Each year, we see steady growth in workshop submissions and attendance, and the research has become more advanced. In 2014, we expect that the workshop (consistent with the eight previous workshops at ACL and NAACL/HLT), will continue to expose the NLP research community to technologies that identify novel opportunities for the use of NLP techniques and tools in educational applications. Topics will include, but will not be limited to, the following:

Invited Speaker

We are happy to have Dr. Norbert Elliot as our invited speaker for the BEA9! Here is the title, abstract and bio.

"Writing Studies and Innovation in Designing NLP Educational Applications: A Multidisciplinary Perspective"

This presentation begins with an examination of Automated Writing Evaluation (AWE) from the perspective of Rhetorical and Composition/Writing Studies. In that multidisciplinary research requires the will to collaborate, the acknowledgement of disciplinary limitations, and the ability to accept risk, the presentation will then turn to a summary of the author's collaborative research with members of the NLP and educational measurement communities. This part of the presentation will focus on mapping broad areas of commonality that yield rich possibilities for collaboration. The third part of the presentation will focus on findings from the past four years of collaborative research. Special attention will be given to the following: relationships between AWE scores and other measures of writing performance, including EPortfolios; and the importance of genre in all writing assessment, with special attention to a recent survey and follow-up interviews conducted with ETS colleagues. The presentation will conclude with a summary of directions for further AWE multidisciplinary research between the NLP and writing studies communities, with special attention to scoring, construct coverage, association with criterion measures, theory-building, and consequence.

Norbert Elliot is professor of English at New Jersey Institute of Technology. He has won both the Conference on College Composition and Communication Outstanding Book Award and the IEEE Professional Communication Society Rudolph J. Joenk Award for outstanding research published in IEEE Transactions on Professional Communication. As advocate of AWE research, Elliot is also co-author of Uses and Limitations of Automated Writing Evaluation Software, a WPA-CompPile Research Bibliography (No. 23, December 2013). He has contributed to the Handbook of Automated Essay Evaluation: Current Applications and New Directions.

As a writing studies researcher committed to wise use of automated writing evaluation (AWE), his work focuses on two areas: promoting multidisciplinary research and innovative applications. To promote collaborative research, in 2013 he co-edited a special issue of Assessing Writing. Focusing on issues ranging from placement to the evaluation of second language writing, reported research included articles from the fields of applied and theoretical linguistics, educational measurement, psychometrics, and writing assessment. In his research on innovative AWE use, Elliot has used the Criterion® Online Writing Evaluation Service at NJIT as a way to rapidly assess the writing ability of admitted first-year students as a way to decrease unwarranted remediation and increase course success. Working with bioengineers and radiologists at NJIT, he is currently working on an experiment to determine if there is a neurological basis for modes of writing discourse.

Submission Information

We will be using the ACL 2014 Submission Guidelines for the BEA9 Workshop this year. Authors are invited to submit a full paper of up to 9 pages of content with up to 2 additional pages for references. We also invite short papers of up to 5 pages of content, including 2 additional pages for references. Please note that unlike previous years, final, camera ready versions of accepted papers will not be given an additional page to address reviewer comments.

Papers which describe systems are also invited to give a demo of their system. If you would like to present a demo in addition to presenting the paper, please make sure to select either "full paper + demo" or "short paper + demo" under "Submission Category" in the START submission page.

Previously published papers cannot be accepted. The submissions will be reviewed by the program committee. As reviewing will be blind, please ensure that papers are anonymous. Self-references that reveal the author's identity, e.g., "We previously showed (Smith, 1991) ...", should be avoided. Instead, use citations such as "Smith previously showed (Smith, 1991) ...".

Please use the 2014 ACL style sheets for composing your paper: (see "ACL 2014 Style Files" section).

We will be using the START conference system to manage submissions:

Important Dates

Presentation Information

Oral Presentations: Long papers accepted for oral presentations are allotted 20 minutes for the talk and 5 minutes for questions. Short papers that are accepted for oral presentations are allotted 15 minutes for the talk and 5 minutes for questions.

Poster Presentations: All papers accepted for a poster presentation will be presented in the session after lunch between 2:00 and 3:30. The posterboards will be self-standing, on top of tables (giving room for laptops, business cards, handouts, etc). The posterboards measure 36 inches high and 48 inches wide. Double-sided tape, pushpins, etc. for affixing the posters to the boards will be provided.

Workshop Program

Download entire BEA9 Proceedings

8:45 - 9:00 Loading in of Oral Presentations
9:00 - 9:15 Opening Remarks
[ paper ] [ slides ]
9:15 - 9:40 Automated Measures of Specific Vocabulary Knowledge from Constructed Responses ("Use These Words to Write a Sentence Based on this Picture")
Swapna Somasundaran and Martin Chodorow
[ paper ] [ slides ]
9:40 - 10:05 Automatic Assessment of the Speech of Young English Learners
Jian Cheng, Yuan Zhao D'Antilio, Xin Chen and Jared Bernstein
[ paper ] [ slides ]
10:05 - 10:25 Automatic detection of plagiarized spoken responses
Keelan Evanini and Xinhao Wang
[ paper ] [ slides ]
10:30 - 11:00 Break
11:00 - 11:20 Understanding MOOC Discussion Forums using Seeded LDA
Arti Ramesh, Dan Goldwasser, Bert Huang, Hal Daume and Lise Getoor
11:20 - 12:30 [Invited Talk] Writing Studies and Innovation in Designing NLP Educational Applications: A Multidisciplinary Perspective
Norbert Elliot
[ slides ]
12:30 - 2:00 Lunch
2:00 - 3:30 BEA9 Poster and Demo Session
2:00 - 2:45 BEA9 Poster and Demo Session A
Natural Language Generation with Vocabulary Constraints
Ben Swanson, Elif Yamangil and Eugene Charniak
[ paper ]
Improving Peer Feedback Prediction: The Sentence Level is Right
Huy Nguyen and Diane Litman
[ paper ] [ poster ]
Similarity-Based Non-Scorable Response Detection for Automated Speech Scoring
Su-Youn Yoon and Shasha Xie
[ paper ] [ poster ]
The pragmatics of margin comments: An empirical study
Debora Field, Stephen Pulman and Denise Whitelock
[ paper ] [ poster ] [ auxiliary slides ]
Syllable and language model based features for detecting non-scorable tests in spoken language proficiency assessment applications
Angeliki Metallinou and Jian Cheng
[ paper ]
Translation Class Instruction as Collaboration in the Act of Translation
Lars Ahrenberg and Ljuba Tarvi
[ paper ]
Automatic evaluation of spoken summaries: the case of language assessment
Anastassia Loukina, Klaus Zechner and Lei Chen
[ paper ] [ poster ]
Demo: An Explicit Feedback System for Preposition Errors based on Wikipedia Revisions [also presented as a poster]
Nitin Madnani and Aoife Cahill
[ paper ]
2:45 - 3:30 BEA9 Poster and Demo Session B
Automated scoring of speaking items in an assessment for teachers of English as a Foreign Language
Klaus Zechner, Keelan Evanini, Su-Youn Yoon, Lawrence Davis, Xinhao Wang, Lei Chen, Chong Min Lee and Chee Wee Leong
[ paper ]
Towards Domain-Independent Assessment of Elementary Students' Science Competency using Soft Cardinality
Samuel Leeman-Munk, Angela Shelton, Eric Wiebe and James Lester
[ paper ]
Sentence-level Rewriting Detection
Fan Zhang and Diane Litman
[ paper ] [ poster ]
Automatic Generation of Challenging Distractors Using Context-Sensitive Inference Rules
Torsten Zesch and Oren Melamud
[ paper ] [ poster ]
ArCADE: An Arabic Corpus of Auditory Dictation Errors
C. Anton Rytting, Paul Rodrigues, Tim Buckwalter, Valerie Novak, Aric Bills, Noah H. Silbert and Mohini Madgavkar
[ paper ]
Surprisal as a Predictor of Essay Quality
Gaurav Kharkwal and Smaranda Muresan
[ paper ] [ poster ]
Demo: Rule-based and machine learning approaches for second language sentence-level readability [also an oral presentation]
Ildikó Pilán, Elena Volodina and Richard Johansson
[ paper ] [ slides ]
3:30 - 4:00 Break
4:00 - 4:25 Exploiting Morphological, Grammatical, and Semantic Correlates for Improved Text Difficulty Assessment
Elizabeth Salesky and Wade Shen
[ paper ] [ slides ]
4:25 - 4:50 Assessing the Readability of Sentences: Which Corpora and Features?
Felice Dell'Orletta, Martijn Wieling, Giulia Venturi, Andrea Cimino and Simonetta Montemagni
[ paper ] [ slides ]
4:50 - 5:15 Rule-based and machine learning approaches for second language sentence-level readability
Ildikó Pilán, Elena Volodina and Richard Johansson
[ paper ] [ slides ]
7:00 - Post-workshop dinner
All are invited, but please let Joel know you are coming so we can get a headcount.

Program Committee

Related Links

Other Upcoming Educational/NLP Events