Conference Highlights – NAACL 2018

After almost a week of jazz, jambalaya, and computational linguistics, Pam and Anselm returned from New Orleans for the NAACL-HLT conference. The main conference event for us was the Tuesday workshop on Building Educational Applications. At the workshop, we presented our contribution to Duolingo’s second-language acquisition modeling (SLAM) shared task. The Duolingo task provided second-language learning data from thousands of users of their app, and posed the challenge of predicting patterns of future translation mistakes in held-out data.

It was great to have a room full of researchers from about a dozen teams who had all focused on the exact same task and data. Burr Settles from Duolingo provided everyone with a careful summary analysis of which approaches worked well (Recurrent Neural Networks, in particular) and a reminder of how vastly second-language learning is spread at a global scale (1.2 billion people are actively learning a second language according to the British Council). By now, the free Duolingo app offers 93 languages, including Esperanto and Klingon. And according to Burr, more people are learning Irish on Duolingo than there are native Irish speakers.

Our lab’s team, which also included David Halpern and recent graduate Alex Rich, finished in 3rd place overall among the fourteen teams that competed. The general architecture we used to model the data was Gradient Boosting Decision Trees (GBDT), supplied with all sorts of engineered features inspired by literature in psychology. We hoped that insights from the large literature on the psychology of learning, memory, and motivation might improve machine learning model predictions. For instance, we engineered word features like the frequency of usage of a word, the usual age that word is learned at, and how much of a cognate a word and its translation are. We also implemented some user features attempting to capture motivation and diligence. You can see our poster and read all about the features we used and our model’s performance here.

pam and anselm at naacl 2018

Although we created a number of features inspired by concepts in psychology, our prediction engine did not specifically model the process of learning. Ultimately, as a psychology lab we want to know whether we can adapt psychological theories of learning for such modeling, and understand human learning behavior better in the process. For the purposes of real-world applications, machine learning algorithms often perform surprisingly well despite being agnostic to theories from the scientific literature in the particular domain. Since such real-world applications are the end goal for many computational linguists, the use of complex machine learning methods has become more and more popular in the field. This trend is prevalent in the field of cognitive science as well: should we sacrifice meaningful, theory-based models for more predictive power (Yarkoni & Westfall, 2017)? Alternatively, how can we unite both machine learning and cognitive science theory to improve performance overall; building better theories that make better models?

At the conference, we met other graduate students and researchers from a variety of backgrounds. We had great conversations with students studying psychology, computer science, or linguistics, as well as with researchers from companies involved in natural language processing, data science, or educational applications like Duolingo. The conference setting of New Orleans was a beautiful backdrop for meeting people while exploring the city’s music and food scene – even as vegetarians we found plenty of delicious food wherever we went. Overall, the Duolingo shared task was fun to compete in, and a great chance to explore an unfamiliar but surprisingly related scientific realm.

Yarkoni, T., & Westfall, J. (2017). Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning. Perspectives on Psychological Science, 12(6), 1100–1122