The 8th Workshop on Innovative Use of NLP for Building Educational Applications

NAACL-HLT 2013 Workshops

Sponsors - Pre-Workshop Information - Workshop Description - NLI Shared Task - Submission Info - Important Dates - Presentation Info - Workshop Program - Program Committee - Related Links - Upcoming Conferences

Conference: NAACL-HLT 2013

Organization: Joel Tetreault (Nuance Communications), Jill Burstein (Educational Testing Service), Claudia Leacock (CTB McGraw-Hill)

Contact Email:

Date : June 13, 2013

Venue : Atlanta, GA, USA


For the first time ever in the history of the workshop, we have solicited sponsorships from major companies working in the fields of NLP and Education. We are pleased to announce that Appen Butler Hill, CTB McGraw-Hill, Educational Testing Service and Pacific Metrics Corporation are all gold level sponsors of the BEA8 workshop!

Pre-Workshop Information

For those attending the BEA8 Workshop this Thursday, June 13, there are a few notes:

Workshop Description

Research in NLP applications for education continues to progress using innovative NLP techniques - statistical, rule-based, or most commonly, a combination of the two. New technologies have made it possible to include speech in both assessment and Intelligent Tutoring Systems (ITS). NLP techniques are also being used to generate assessments, and tools for curriculum development of reading materials, as well as tools to support assessment and test development. As a community, we continue to improve existing capabilities and to identify and generate innovative and creative ways to use NLP in applications for writing, reading, speaking, critical thinking, and assessment.

In 2012, the use of NLP in educational contexts took two major steps forward. First, outside of the computational linguistics community, the Hewlett Foundation reached out to both the public and private sectors and sponsored two competitions: one on automated essay scoring (Automated Student Assessment Prize: ASAP, Phase 1), and a second on short-answer scoring (Phase 2). The motivation driving these competitions was to engage the larger scientific community in this enterprise. The two competitions were inspired by the Common Core State Initiative, an influential set of standards adopted by 45 states in the U.S. The Initiative describes what K-12 students should be learning with regard to Reading, Writing, Speaking, Listening, and Media and Technology. Another breakthrough for educational applications within the computational linguistics community was the second edition of the “Helping Our Own” grammatical error detection/correction competition at last year’s BEA workshop – where 14 systems competed. In 2013, independent of the BEA workshop, there will be two shared task competitions. This year’s CoNLL Shared Task is on grammatical error correction and there is a SemEval Shared Task on Student Response Analysis. Both of these competitions will increase the visibility of the educational problem space in the NLP community.

In this year’s BEA workshop, we are soliciting papers across a broad range of educational applications, including: intelligent tutoring, learner cognition, use of corpora, grammatical error detection, tools for teachers and test developers, and automated scoring and evaluation of open-ended responses. Since the first workshop in 1997, “Innovative Use of NLP in Building Educational Applications” has continued to bring together all NLP subfields to foster interaction and collaboration among researchers in both academic institutions and industry. The workshop offers a venue for researchers to present and discuss their work in these areas. Each year, we see steady growth in workshop submissions and attendance, and the research has become more innovative and advanced. In 2013, we expect that the workshop (consistent with previous workshops at ACL 1997, NAACL/HLT 2003, ACL 2005, ACL 2008, NAACL HLT 2009, NAACL HLT 2010, ACL 2011, NAACL HLT 2012), will continue to expose the NLP research community to technologies that identify novel opportunities for the use of NLP techniques and tools in educational applications. At NAACL HLT 2012, the workshop coordinated with the HOO shared task for grammatical error detection, generating a much larger poster session that was lively and well-attended. In 2013, the workshop will host a Native Language Identification Shared Task.

The workshop will solicit both full papers and short papers for either oral or poster presentation. Given the broad scope of the workshop, we organize the workshop around three central themes in the educational infrastructure:

  1. Development of curriculum and assessment (e.g., applications that help teachers develop reading materials)
  2. Delivery of curriculum and assessments (e.g., applications where the student receives instruction and interacts with the system);
  3. Reporting of assessment outcomes (e.g., automated scoring of open-ended responses)

Topics will include, but will not be limited to, the following:

NLI-2013 Shared Task

We are pleased to host the first edition of a shared task on Native Language Identification (NLI). The shared task will be organized by Joel Tetreault, Aoife Cahill and Daniel Blanchard. NLI is the task of identifying the native language (L1) of a writer based solely on a sample of their writing. The task is typically framed as a classification problem where the set of L1s is known a priori. Most work has focused on identifying the native language of writers learning English as a second language. To date this topic has motivated several ACL and EMNLP papers, as well as a master’s thesis.

Native Language Identification (NLI) can be useful for a number of applications. In educational settings, NLI can be used to provide more targeted feedback to language learners about their errors. It is well known that learners of different languages make different errors depending on their L1s. A writing tutor system which can detect the native language of the learner will be able to tailor the feedback about the error and contrast it with common properties of the learner’s language. In addition, native language is often used as a feature that goes into authorship profiling, which is frequently used in forensic linguistics. Details on the shared task can be found on the website:

Submission Information

We will be using the NAACL-HLT 2013 Submission Guidelines for the BEA-8 Workshop this year. Authors are invited to submit a full paper of up to 8 pages in electronic, PDF format, with up to 2 additional pages for references. We also invite short papers of up to 4 pages, including 2 additional pages for references. Papers which describe systems are also invited to give a demo of their system. If you would like to present a demo in addition to presenting the paper, please make sure to select either "full paper + demo" or "short paper + demo" under "Submission Category" in the START submission page.

Previously published papers cannot be accepted. The submissions will be reviewed by the program committee. As reviewing will be blind, please ensure that papers are anonymous. Self-references that reveal the author's identity, e.g., "We previously showed (Smith, 1991) ...", should be avoided. Instead, use citations such as "Smith previously showed (Smith, 1991) ...".

Please use the 2013 NAACL-HLT style sheet for composing your paper: (see Format section for style files).

We will be using the START conference system to manage submissions:

Please note that we are using the same submission page for both the BEA-8 and the NLI-2013 Shared Task. When submitting to the NLI-2013, please select "NLI 2013 Shared Task" under "Submission Type."

Important Dates

Presentation Information

Oral Presentations: Long papers accepted for oral presentations are allotted 20 minutes for the talk and 5 minutes for questions. Short papers that are accepted for oral presentations are allotted 15 minutes for the talk and 5 minutes for questions.

Poster Presentations: All papers accepted for a poster presentation (this includes NLI Shared Task papers) will be presented in the session after lunch. We will divide the poster session so that people presenting posters will have a chance to see other posters and don't have to be at their poster the entire time.

The posterboards will be 3 ft high and 4 ft wide, slightly bending about 10 in in at both left and right ends. They will be set on top of 4-5 ft long, 18 in deep tables so there will be room to also hold handouts, business cards, computer, etc. There will also be one chair will be at each table. Double-sided tape, pushpins, etc. for affixing the posters to the boards will be provided.

Workshop Program

[ BEA8 Proceedings ]
8:45 - 9:00 Loading in of Oral Presentations
9:00 - 9:15 Opening Remarks
[ paper ] [ slides ]
9:15 - 9:40 The Utility of Manual and Automatic Linguistic Error Codes for Identifying Neurodevelopmental Disorders
Eric Morley, Brian Roark and Jan van Santen
[ paper ] [ slides ]
9:40 - 10:05 Shallow Semantic Analysis of Interactive Learner Sentences
Levi King and Markus Dickinson
[ paper ] [ slides ]
10:05 - 10:30 Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English
Daniel Dahlmeier, Hwee Tou Ng and Siew Mei Wu
[ paper ] [ slides ]
10:30 - 11:00 Break
11:00 - 11:25 Developing and trialling a self-assessment and tutoring system
O Andersen, H Yannakoudakis, F Barker and T Parish
[ paper ] [ slides ]
11:25 - 11:45 Automated Essay Scoring for Swedish
Robert Östling, André Smolentzov, Björn Tyrefors Hinnerich and Erik Höglin
[ paper ] [ slides ]
11:45 - 12:10 A Report on the First Native Language Identification Shared Task
Joel Tetreault, Daniel Blanchard and Aoife Cahill
[ paper ] [ slides ]
12:10 - 1:50 Lunch
1:50 - 2:40 BEA8 and NLI Poster Session A
Role of Morpho-Syntactic Features in Estonian Proficiency Classification
Sowmya Vajjala and Kaidi Loo
[ paper ] [ poster ]
Automated Content Scoring of Spoken Responses in an Assessment for Teachers of English
Klaus Zechner and Xinhao Wang
[ paper ]
Applying Unsupervised Learning To Support Vector Space Model Based Speaking Assessment
Lei Chen
[ paper ]
VTEX System Description for the NLI 2013 Shared Task
Vidas Daudaravicius
[ paper ]
NLI Shared Task 2013: MQ Submission
Shervin Malmasi, Sze-Meng Jojo Wong and Mark Dras
[ paper ]
Cognate and Misspelling Features for Natural Language Identification
Garrett Nicolai, Bradley Hauer, Mohammad Salameh, Lei Yao and Grzegorz Kondrak
[ paper ]
Maximizing Classification Accuracy in Native Language Identification
Scott Jarvis, Yves Bestgen and Steve Pepper
[ paper ]
Simple Yet Powerful Native Language Identification on TOEFL11
Ching-Yi Wu, Po-Hsiang Lai, Yang Liu and Vincent Ng
[ paper ]
NAIST at the NLI 2013 Shared Task
Tomoya Mizumoto, Yuta Hayashibe, Keisuke Sakaguchi, Mamoru Komachi and Yuji Matsumoto
[ paper ]
Exploring Syntactic Representations for Native Language Identification
Ben Swanson
[ paper ]
Feature Space Selection and Combination for Native Language Identification
Cyril Goutte, Serge Léger and Marine Carpuat
[ paper ]
Recognizing English Learners' Native Language from Their Writings
Baoli LI
[ paper ]
Experimental Results on the Native Language Identification Shared Task
Amjad Abu-Jbara, Rahul Jha, Eric Morley and Dragomir Radev
[ paper ]
Discriminating Non-Native English with 350 Words
John Henderson, Guido Zarrella, Craig Pfeifer and John D. Burger
[ paper ]
Identifying the L1 of non-native writers: the CMU-Haifa system
Yulia Tsvetkov, Naama Twitto, Nathan Schneider, Noam Ordan, Manaal Faruqui, Victor Chahuneau, Shuly Wintner and Chris Dyer
[ paper ] [ poster ]
2:40 - 3:30 BEA8 and NLI Poster Session B
Automated Scoring of a Summary-Writing Task Designed to Measure Reading Comprehension
Nitin Madnani, Jill Burstein and John Sabatini
[ paper ]
Prompt-based Content Scoring for Automated Spoken Language Assessment
Keelan Evanini, Shasha Xie and Klaus Zechner
[ paper ] [ poster ]
Inter-annotator agreement for dependency annotation of learner language
Marwa Ragheb and Markus Dickinson
[ paper ] [ poster ]
Improving Native Language Identification with TF-IDF Weighting
Binyam Gebrekidan Gebre, Marcos Zampieri and Peter Wittenburg
[ paper ]
Using N-gram and Word Network Features for Native Language Identification
Shibamouli Lahiri and Rada Mihalcea
[ paper ] [ poster ]
Combining Shallow and Linguistically Motivated Features in Native Language Identification
Serhiy Bykh, Sowmya Vajjala, Julia Krivanek and Detmar Meurers
[ paper ] [ poster ]
Native Language Identification: a Simple n-gram Based Approach
Binod Gyawali, Gabriela Ramirez and Thamar Solorio
[ paper ]
The Story of the Characters, the DNA and the Native Language
Marius Popescu and Radu Tudor Ionescu
[ paper ] [ poster ]
Using Other Learner Corpora in the 2013 NLI Shared Task
Julian Brooke and Graeme Hirst
[ paper ] [ poster ]
Native Language Identification: A Key Ngram Category Approach
Kristopher Kyle, Scott Crossley, Jianmin Dai and Danielle McNamara
[ paper ]
Linguistic Profiling based on a Wide Range of Features and Native Language Identification
Andrea Cimino, Felice Dell'Orletta, Giulia Venturi and Simonetta Montemagni
[ paper ]
LIMSI's participation to the 2013 shared task on Native Language Identification
Thomas Lavergne, Gabriel Illouz, AurÁlien Max and Ryo Nagata
[ paper ]
Feature Engineering in the NLI Shared Task 2013: Charles University Submission Report
Barbora Hladka, Martin Holub and Vincent Kriz
[ paper ]
Native Language Identification with PPM
Victoria Bobicev
[ paper ]
Native Language Identification using large scale lexical features
André Lynum
[ paper ]
3:30 - 4:00 Break
4:00 - 4:20 Evaluating Unsupervised Language Model Adaptation Methods for Speaking Assessment
Shasha Xie and Lei Chen
[ paper ] [ slides ]
4:20 - 4:40 Improving Interpretation Robustness in a Tutorial Dialogue System
Myroslava Dzikovska, Elaine Farrow and Johanna Moore
[ paper ] [ slides ]
4:40 - 5:00 Detecting Missing Hyphens in Learner Text
Aoife Cahill, Martin Chodorow, Susanne Wolff and Nitin Madnani
[ paper ] [ slides ]
5:00 - 5:20 Applying Machine Translation Metrics to Student-Written Translations
Lisa Michaud and Patricia Ann McCoy
[ paper ] [ slides ]
5:20 - 5:30 Closing Remarks
7:00 - Post-workshop dinner at Haveli Indian Cusine at 255 Spring Street, Atlanta.
[ Directions from NAACL conference site to Haveli ]
All are invited, but please let Joel know you are coming so we can get a headcount.

Program Committee

Related Links

Other Upcoming Educational/NLP Conferences