Translating Cognitive Science to EdTech: Searching for Impact

I’m a post-doc in the Gureckis lab working on translating some of the ideas in cognitive science in educationally-relevant directions. I’ve always felt it is a shame that despite thousands of studies on learning and memory most cognitive psychology findings are not translated or absorbed into educational practice (I’m not alone in this). In this post, I will describe my own journey in the lab to bridge this gap. My next post will introduce in more detail the types of applications I am developing and discuss some of the design decisions that need closer scientific scrutiny. For now I am planning to just share our progress and the adventure of taking cognitive science out of the lab and into the real world! I’d be really interesting in hearing other people’s thoughts about translating cognitive science and education, so feel free to comment below!

Just after I arrived at NYU in January 2015, I told Todd that I had an ambition to enter the Global Learning XPRIZE (GLXP), with the goal of creating tablet software that would teach children basic literacy and numeracy skills. The competition, put forth by the same XPRIZE Foundation that kickstarted the private space industry in 2004, will award $1 million to up to five finalists who will be chosen this November after their software is tested by a panel of judges. The software from each finalist will then be loaded onto 400 tablets and distributed to 7- and 8-year-olds in Tanzanian villages (one per child, 10 per village) who have enrolled and been given math and literacy pre-tests. The kids will be given no guidance on how or when to use the tablets, and indeed may not have ever had any formal schooling. After 1.5 years of using the tablets (which will be charged daily via solar charging stations, but without internet), the children will take a post-test, and based on their improvement a single grand prize winner will be selected and awarded $10 million. Ambitious? Certainly. But possible? Perhaps.

The Global Learning XPrize represents the ultimate application that I believe cognitive scientists should be interested in, but often merely pay lip service to.

Creating a science-based entry to the GLXP has been my post-doc side project while I continued to design, implement, run, analyze, and write up more traditional cognitive psychology experiments. To me, the GLXP represents the ultimate application that I believe cognitive scientists should be interested in, but often merely pay lip service to. With my background in language acquisition, and our mutual interest in active, self-directed learning, I thought Todd and I would have as good a shot as anyone at designing, implementing, and improving automated tutoring software. At the very least, I wanted him to know that I had an eye towards (ambitious) applications of the scientific knowledge garnered in our research. With this in mind, Team egoTeach was born and officially registered, and along with ~130 other teams from around the globe.

In brief, the egoTeach approach is to 1) create a suite of fun games that continuously present and test 2) shared educational content between them, while 3) continuously adapting to the learner’s performance by scaling up (or down) game dynamics as well as content difficulty. If this sounds like a lot of other edTech products (albeit on a grander scale), that shouldn’t be surprising: presenting concepts in multiple motivating formats, scaffolding content, and adapting to the learner’s pace are clearly desirable features. The difference is that egoTeach bakes in a lot of hard-earned wisdom gleaned from decades of cognitive psychology experiments on memory and learning. However, this is far from easy due to the way we do cognitive science: experiments typically manipulate one (maybe two) factor and attempt to control for all others. In contrast, engineering demands that we create a complete product, necessitating us choosing the right recipe of factors and their several interactions.

egoTeach bakes in a lot of hard-earned wisdom gleaned from decades of cognitive psychology experiments on memory and learning.

I can expand on our specific difficulties in a later post, but for now I will illustrate with a simple example: imagine you wake up at 9am in a panic, suddenly remembering that you have a French vocabulary test at 10am that you forgot to study for. Given one hour, how should you distribute your study time over the, say, 60 words you expect to be on the quiz?

You could simply stare at each French-English word pair for one minute, but you recall that decades-old findings from memory studies indicate that it is better to study each item multiple times (even for an equal amount of total study time). So you decide to cycle through the list three times (or four? five? surely not 50?), studying each item for 20 seconds. However, looking at the first few words you realize you already know some of them quite well, and you get bored before the allocated study time is done: but should you trust this feeling? Judgements of learning in memory studies are correlated with later performance, but not perfectly. Nonetheless, active memory studies (e.g. this one) have shown that learners benefit from being able to control how long they study each item, so it may be better to move on when you feel you’re ready. You may not even need to study the easy ones as often; your time may be better spent on the hard ones. However, if your memory is poor or if there are too many items, maybe you should only try to learn a portion of them: but how many is too much? Another problem with going down the list repeatedly is that you will learn the sequence, much like you will learn to expect the next song on your favorite album as the last one finishes. Recall for individual items may depend in part on having the sequential context, so it would be better to randomize the order to some extent. Of course, that creates a logistical nightmare: unless you have study software or a flash card system that can aid you, keeping track of the how often each item should be seen–and when, based on its difficulty and your overall performance–is impossible. After all, you can’t spend your entire hour trying to think up, let alone implement such a system! For the French test problem, there are in fact a number of spaced repetition apps out there to help you study, but what about fundamental concepts essential to building literacy and numeracy in the first place?

As it turns out the more we know about the cognitive science, the harder it may be to identify the best application of that knowledge! This is the main challenge that we have been addressing in our GLXP submission. In particularly, Hugo Goulart de Lucena and I have been working on a series of games to teach the alphabet, spelling, numbers, counting, addition, and subtraction using more than just spaced repetitions of rote facts.

Hugo and George at the D.C. Summit
As an initial update on this adventure, on Friday, April 1 Hugo and I were in Washington, D.C. at the World Bank HQ for a GLXP summit meeting. We were excited to meet the other teams, eager to see their presentations, and giddy that the GLXP was beginning to feel real–and that we were on track to be a part of it! After a security check (apparently 5 heads of state were in and out of the building that day for the nuclear summit), our expectations for the day soared as we walked into the high-ceilinged conference room, with it’s immense oval conference table and personal microphone/translation stations. Each of the eleven teams in attendance (two more arrived later) gave a brief introduction. I was relieved to hear that we were not the only team with academics: teams affiliated with CMU (RoboTutor) and UCSB–both headed up by computer science professors–were there, but I was at least as curious to see the ideas the other teams had come up with. Below is a brief characterization of each team.

  • AutoCognitiva, with engineers in Hong Kong and an educator in the US, presented a nicely-designed collection of ‘sandboxes’ that implement a constructivist style of education, allowing children to actively experiment (and fail) with, for example, phonic spelling (using a specially-designed phonics keyboard), subject-verb-object sentence construction (enforcing correct structure, but not meaning), and a number line using blocks to teach quantity and addition that I particularly liked. Although these sandboxes were far less game-like than our egoTeach apps, AutoCognitiva’s approach resonated most strongly with ours, given that we both focus on exploration and active learning to some extent. However, our games do have goals — even if the child has to explore to discover them (or maybe they will come up with their own goals!).
  • RoboTutor, founded by Prof. Jack Mostow of CMU, comes from a decades-old literacy project of his that uses quite stilted speech to teach concepts and ask kids questions. It seemed to me a rote repetition form of teaching, but although I personally dislike this it may work. The math approach is to give narrated math problems, with robot-narrated hints when mistakes are made.
  • GLEN, founded by Prof. Madhow from UC Santa Barbara, is mostly focused on teaching English–but will grudgingly do a Swahili translation to be eligible for the GLXP.
  • Narrativas is a one-man show with a rather abstract and arcane timeline idea, and is admittedly at the idea stage but in search of a technical co-founder.
  • John Harrison Project is run by Patrick, a self-described outsider with math and CS training who was intentionally vague about his approach, but tellingly gave us all an overview on state-of-the-art Swahili text-to-speech and speech recognition. Although actively-developed open source solutions such as PocketSphinx and Kaldi are making speech recognition easier than ever, implementing (and testing) grammar specifications for Swahili is a challenge. I assume the John Harrison project will be attempting to read children’s books aloud, perhaps prompting children to read along. Patrick also admitted he is more focused on the Adult Literacy X-prize.
  • LearnLeapFly are a duo of Canadian math PhDs who gave a well-presented talk that said very little about what they are actually doing, although they claimed to soon be open-sourcing the development of their game engines and authoring tools from their 20-person team sometime — I look forward to seeing it!
  • EduApps4All had the most heartfelt story, for the founder Isabelle has persevered while working towards the GLXP’s goal for 10 years–before it was a twinkle in XPRIZE’s eye! EduApps4All has a few games with happy, cartoony graphics and quite a bit of content. It looks as though the interactions are mostly limited to touching fixed buttons on the screen, which is rather different than our approach of making integrating the response dynamics with the stimuli.
  • DeepOrigins didn’t show up in time for introductions, but I had a nice chat with two of them at lunchtime and was intrigued to hear that their approach uses Ayurvedic music to teach all of the content. I’m not sure what that means, exactly, but I’m very curious!
  • Sleeping Dragons – Masako and Kenneth didn’t do a full presentation, but they mentioned they are linked with Tulane and seemed quiet and competent: I wish I’d had more of a chance to speak with them.
  • Curriculum Content International, much like their name suggests, is a company that licenses content to large textbook publishers–and has done so for 30 years. The leader, Reginald Poe, admitted that they do not have the technical expertise to create the apps, but was looking to partner with a team that needed content.
  • Dev4X is another NY-based team, founded by seasoned entrepreneurs Bodo and Michael, that is creating a content-agnostic platform with the hopes of attracting content companies. It looks like they will then try to sequence/weave the content together. Dev4X is notable for trying a radically open approach: anybody who wants to volunteer can be involved simply by joining their weekly Skype calls.
  • Linguaculturalists are a family team (Michael and Michelle Hall, I believe) making what seems to be Farmville for education: you plant seeds that grow letters, which can be sold at the market for in-game money, or perhaps combined to build words. They are to be commended for actually publicly posting their code (and for actually having code!). I’m eager to see if their approach is successful: I do think such an ambitious game is exciting, but I think it will be quite difficult to effectively adapt a curriculum for this style of presentation…but I wish them luck! It would be truly exciting if it works.
  • Library4All is a pre-existing organization that is entirely focused on the literacy side. They certainly have content but I did not manage to hear from them how they will teach the alphabet and spelling.
  • Midnight Illusions is, I believe, the only team of professional app developers that was at the D.C. summit. Coming from outside Toronto and represented by Jason Bavington, they seem sharp, although we didn’t see their full presentation because they had already given it in Paris! I look forward to reviewing that video.
  • Todo School has a UC Berkeley child development researcher involved, but I’m afraid we didn’t get any details (and their website is down).

Hugo in World Bank conference room
In summary, teams are taking fairly varied approaches, ranging from rote memorization to games to explorable sandboxes. Although academia had some representation, no other cognitive psychologists were evident. Nonetheless, Hugo and I came away energized: we still felt like our concepts were among the best, and that we were on track to submit a complete solution. Since the D.C. summit meeting, Hugo and I have been working hard on our apps, which I will introduce in the next installment. Thanks for tuning in!

Tags: , ,

  1. Hello! Upon reading your French test example of how you and Hugo came about creating this app, I was wondering if you had taken into account/know a way of bringing in AI to learn the child’s abilities – via reaction time, their successes in different difficulty levels, etc. This way, the app can not only provide the information to the child, as well as scaffold them up to more solidified, confident learning, but also, say you were to expand the app to various versions to help older students (or college students who do wake up in the middle of the night panicking before an exam), would there be a feature that will stop your studying after a certain amount of time or ‘flips’ through the words/games? You wrote “However, if your memory is poor or if there are too many items, maybe you should only try to learn a portion of them: but how many is too much?” and I am curious is the app could dictate when enough is enough.

    In other words, after using the app a good amount of times through games, flashcard-style learning, etc., would the app be able to ‘learn’ how much information my mind can consumer in one sitting and maybe warn me when I get to a point of overload or too much information that may actually harm my studying, causing me to backtrack?

    Of course this is a very complex concept and I have not spelled out the logistics of said proposal, though I think it would be an interesting conversation of brining in this aspect to the app.

    If children were to use this app, play a series of games and start out by getting 1 out of 10 right, then 2 out of 10, then 5 out of 10, then 8 out of 10 then 2 out of 10 again, the app could store that information as “backtracking” and take note that at the level, the child can only understand x amount of information, or on the other side of this, can only take 5 minutes (or however long it took before they backtracked) before they stop taking in the information. In a much less specific explanation, the app would become personalized to that child’s education – the physical way that their brain takes in information. I recall you mentioned that the app would scale up or down in difficulty level and go at the pace for the student, but is there an aspect that will be specific to the learner or will it be based off of a generic level of “1, 2, 3, 4, 5…” that will fit each student into a category that is closest to them? This is a fantastic component of the app, vital I believe, though many (currently more outspoken) issues with American education is treating every mind as a carbon copy learner (hence why so many educators dislike the common core) and the material isn’t catered towards that learner and many fear the child’s difficulty in studying will stray them away from their education (which you also touched on as well in your French exam example).

    In addition to this, you wrote “As it turns out the more we know about the cognitive science, the harder it may be to identify the best application of that knowledge! This is the main challenge that we have been addressing in our GLXP submission. In particularly, Hugo Goulart de Lucena and I have been working on a series of games to teach the alphabet, spelling, numbers, counting, addition, and subtraction using more than just spaced repetitions of rote facts.” Does the app have a series of videos, hands on, written and verbal exercises? I understand the early stages of the app are to teach (illiterate?) children basic literacy, though in my experience with education, each person learns in a single or combination of ways: hands on, visual, verbal, reading or writing. I’m sure there are more – but it may be interesting to include a study from maybe a cognitive or developmental psychologist about how children learn or how their brains develop under different exposures to different tactics of initially learning and then retaining that information. You could apply that information to your app and teach the app to further personalize the app to penetrate the child’s brain even more deep-seeded!

    Also to further integrate the information they’re learning, children need to have it reinforced in their lives (I’m not citing any study this is just a general idea). Will the app have incentives to go from one level to the next and will the children be given words, themes, etc. that is prevalent in their society or our Westernized society? Language as well.

    I also was wondering if you could (I mean, after the competition is over, I’m sure you don’t want to divulge all your secrets just yet) explain the functions of your app on this website? I’m having a difficult time opening it on your website – however I did get a chance to look over the graphs and I’ll be honest I didn’t quite understand all of them (I work in fashion and entertainment, my brain only works so far into psychology haha) but what I did were helpful in understanding your process in creating the app.

    Once again, this is all just hypothetical. I’m sure you’ve considered this and more! This is coming from a very much less educated psychology undergraduate student! I’m not asking a question or asking for a response, it’s just a thought. Good luck! I will stay updated on your progress, as I believe this is a great idea!

    Kate Ramsay
  2. Hi Kate! Thanks for the interest and your detailed comment–and I’m sorry for my slow reply! You’ve touched on many important issues that we are indeed thinking about, and some of which we have implemented already! First of all, you’re entirely correct that the apps should (and do already, to an extent) use the learner’s responses (time and correctness) to adjust the difficulty, not only of the game dynamics (e.g., number of options and speed of each trial), but also of the content. Although we are considering testing it in the future, we don’t yet entirely lock out content, but rather re-prioritize each stimulus continuously: if you get a stimulus (e.g., word, number, or letter) correct quickly, it will not be tested again for a while. If you get one wrong, it will be shown again sooner, and in a new context. We’ve just finished a pilot study in Tanzania where we allowed children to self-regulate which games (and thus to large extent which content) they experienced. After we understand how well they distribute their attention to what they should be learning, we will consider restricting their choices if their choices are suboptimal (e.g., blocking them from the alphabet game if they already know it well, forcing them to move onto spelling words). By studying how they behave in freeplay, we hope to gain more insights into 1) what a successful learner looks like, and 2) how we can guide less successful learners to improve. We will use the data the trial-level data we collect to do most of the things you mention. More details will be posted soon both here and on our other blog (where a youtube video of the latest apps is also available!):

    For now at least, we do not include any rote lectures: we are trying instead to introduce concepts embedded in interactive playgrounds, with the rules of the games being entirely consistent with the to-be-learned concepts. However, we acknowledge that at some points short lectures seem to be an efficient means of conveying knowledge–if done at the appropriate level of knowledge, and for not too long a duration!

    Finally, you’re right that the important thing is for kids to integrate their new knowledge into their daily lives: we believe much of this will happen naturally (e.g., they will want to read things and will realize they now can), but we are also working on a storytelling app that prompts them to think about, write and illustrate stories. If you can think of other ways to use the tablets to encourage kids to engage their knowledge in their outside life, please let us know! Thanks again for the interest!

    George Kachergis