One really nice thing about the Arudino platform is that it provides a +5V power source as a user-accessible pin. This can be used to power up user-made circuits without a heavy power source (i.e., separate batteries or a signal generator). However, many circuit components require both a positive and a negative voltage source to function properly. One example is an op-amp which typically requires a +VCC and -VCC. Is there a way to convert the single +5V source on the Arduino into a -5V source?

If you are trying to cut costs, taking two standard 9V batteries and connecting them in series will give you a +9V and a -9V. However, the problem with that solution is the bulk of two large batteries simply to power your op-amp (not good on a mobile robot!). If you can shell out a couple dollars you can pick up a tiny integrated circuit (IC) chip that acts as a voltage inverter (the ICL7660). Mouser electronics sells them here for a couple bucks although there may be cheaper versions/brands.

All you need for this circuit is the ICL7760 and two 10uF (micro Farad) capacitors (also available at Mouser if you search). The circuit diagram looks like this:

A final wired up version looks like this on a small bread board:

Basically you feed in the +5V source from the Arduino board into the top right pin of the ICL7660 (looking with the half-moon depression oriented up, the red wire in my photo). You wire up a couple places to ground on the Arduino board (black wires in my photo), place the capacitors (check the orientation on these!), and you will get negative voltage out of the bottom right pin that is roughly the same magnitude as the input at the top (i.e., coming off the yellow wire). The only downside is that, due to the ICL7660, this circuit can’t provide much current without effecting the voltage. Thus, it may not work for all applications, particular amps designs that draw a lot of power.

Thanks to this forum for many helpful tips. I simply translated here the solution that worked well for me into the language we’ve used in my class and posted it here for reference.

I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person–to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know the the real difference was the thing she thought she had under control.

She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happened.”

- Richard Feynman “Cargo Cult Science“

The trip also gave us the opportunity to spend some time with Alex Rich, who will be joining the lab this fall (welcome!!). We are all excited about starting to work with him.

Unfortunately, on our way back to New York we got caught in what felt like a minor snow blizzard which had us crawl back to the city at a painstakingly slow speed. A rewarding experience nevertheless, and we hope to do this again next winter!

Group shot at the top of the mountain

Where is everyone?

Clearly, Doug's been doing this all his life

During the summer after my freshman year I wanted to find an internship at a cognitive science lab to try and find out first hand what its like to work in the field. Professor Gureckis’ affiliations were cognition and perception and his lab computation and cognition seemed really interesting. The lab mainly researches cognitive processes like memory, learning and decision-making. I emailed professor Gureckis asking if I could work at his lab over summer and he let me work with him for 8 weeks. It was really quite inspiring to sit in on their lab meetings and see for myself how the graduate students and the professors, discussed experiments, made inferences, critiqued each others ideas and made futures plans on what to do next. The lab nurtures a very productive environment with a really friendly atmosphere. I got along well with everyone within the first few days of working at the lab and the faculty and students seem to understand each other quite well.

Professor Gureckis wanted to combine two of his passions, robotics and psychology, and create an environment for students to conduct psychology experiments using robots to understand the two areas better – the science of experimentation in psychology and a peek into robotics. This course would give the students a taste of what its like to reverse engineer the human mind by experimenting with robot subjects and reverse engineering their inner workings. To help him create this course, I helped find the required robot, assembled it and created a couple of programs to perform certain tasks. He provided me with academic papers that dealt with similar topics and a book by Valentino Braitenberg called ‘vehicles’ which I used to create programs for the robot to execute. This was my first proper introduction to robotics and it was really nice to explore it on my own using the libraries, the internet and shops like RadioShack. I finally assembled a robot with an Arduino microchip and programmed it to perform simple maneuvers, respond to sound and calculate small distances.

As a final part of my project, I gave a sample class to the Cognition summer class. Under professor Gureckis’ guidance I gathered the required reading material for the class and the created the required handouts for the experiment we were going to conduct on that day. The students tried to reverse engineer the light seeking robot and find out how the behavior of the robot emerged from the simple wiring of the circuitry.

Overall, it was an amazing learning experience that involved diving into a new field, a lot of hands-on work and a bit of programing. It gave me the much-needed taste of some multidisciplinary work.

Todd was lucky to not lose power, but the view from his window showed the changes to the West Village pre-/post-storm:

Anna experienced the East River rising up and flooding into her building at Stuyvesant Town.

John watched the whole thing from his rooftop in Williamsburg where he grabbed a great shot of Manhattan bathed in artificial darkness.

Anyway, although it seems like things are starting to return to some sense of normalcy in the city, many are still being seriously affected along the New Jersey and New York shoreline. Hang in there everyone!

We recently published an article in Perspectives in Psychological Science arguing that one such area of confluence may be the study of active or self-directed learning.

One of the most influential and long-standing ideas in education is that students learn more effectively when they are “active” or have some control over the learning process (e.g., a hands-on lab class). This is often placed in opposition to more “passive” forms of learning (e.g., sitting in a lecture hall).

One reason being self-directed may improve learning is that it allows individual learners to focus their study effort on parts of the world they have not yet mastered. Rather than wasting time on material they already know, self-directed learners can keep pushing the boundaries of their knowledge. The end result is more total learning, since less time is spent on redundant material.

Over the last decade or so a similar idea has been making waves in the machine learning community under the name “active learning“. Here the main goal is to speed up the training of a machine learning system, but the deeper issue is ultimately the same one faced by human learners. Just as a student might choose to ignore material they have already mastered, it makes sense for machine learning systems to know what kinds of data are likely to be informative and focus effort on those expected to be particularly revealing.

Better Learning from Less Data

How can you know how informative something will be before you’ve learned about it? Algorithms developed in the machine learning literature solve this problem by estimating how much learning will occur given different possible answers to a question the learner asks. For example, a recommendation engine (like Netflix or Amazon) might aim to predict whether you like lots of different products, but it can only ask you to rate a very small proportion of items before it becomes annoying. As a result, the websites should only ask about an item if it is expected to lead to a better understanding of your general preferences (regardless of whether you say you like that particular product or not). Given the choice, perhaps knowing if you like or dislike Kubrick’s art-house classic 2001: A Space Odyssey is better for predicting your other movie preferences than knowing if you like a summer block-buster like Transformers.

The exciting thing about this research (at least to cognitive scientists like us) is that it offers a precise, quantitative framework for understanding self-directed learning in humans. In effect, we can “borrow” ideas from the machine learning literature to help us develop theories about how people choose to collect information while they learn. Somewhat serendipitously, some of the algorithms we “borrow” from machine learning actually can predict how humans will gather information when learning (e.g., [1, 2, 3])! It’s also possible for the flow of ideas to run in reverse: ideas and findings in psychology may someday help inform better machine learning systems.

Our paper reviews the psychological and cognitive factors that determine whether self-directed learning will be an effective learning strategy in humans (which it may not be for certain kinds of learning problems). We then try to find some common ground between the formal methods developed in machine learning research and our understanding of self-directed learning from psychology.

Check out the paper here. For those interested in learning more about this topic, there are a lot of great references to both the machine learning and cognitive science literatures contained within. Also check out our paper archive for some of our recent work on this.

From the abstract:

A widely advocated idea in education is that people learn better when the flow of experience is under their control (i.e., learning is self-directed). However, the reasons why volitional control might result in superior acquisition and the limits to such advantages remain poorly understood. In this article, we review the issue from both a cognitive and computational perspective. On the cognitive side, self-directed learning allows individuals to focus effort on useful information they do not yet possess, can expose information that is inaccessible via passive observation, and may enhance the encoding and retention of materials. On the computational side, the development of efficient “active learning” algorithms that can select their own training data is an emerging research topic in machine learning. This review argues that recent advances in these related fields may offer a fresh theoretical perspective on how people gather information to support their own learning.

Handy Bibtex reference:

@article{Gureckis-2012-PPS,
Author = {Gureckis, Todd M. and Markant, Douglas B.},
Doi = {10.1177/1745691612454304},
Journal = {Perspectives on Psychological Science},
Number = {5},
Pages = {464-481},
Title = {Self-Directed Learning: A Cognitive and Computational Perspective},
Volume = {7},
Year = {2012},
}

For people who missed the workshop (or attendees who would like to review), we’ve collected up the resources that we presented here. Hopefully it can jog your memory if you’ve forgotten something a few months down the line.

Mechanical Turk/psiTurk

The first set of presentations were given by Todd Gureckis and John McDonnell covering Amazon’s Mechnical Turk and, in particular, some software we have shared on GitHub for helping you to develop these types of experiments in your own work.

• Our HTML slides are hosted here. You can skip directly to the information about psiTurk by clicking here
• More information about psiTurk is also available on the psiTurk GitHub site (see the README.md and Wiki for more information)

To get psiTurk to work, at minimum, you need the following software installed on your computer.

• If you’ve never used Python before, we recommend you use the Enthought Python distribution, because it comes with easy_install, a tool for easily installing Python packages. The academic version of Enthought can be found here: http://www.enthought.com/repo/.epd_academic_installers
• Once you’ve installed Enthought, you should be able to open a terminal window and install Flask (the web server library we’ll be using), SQLAlchemy (the database manipulation package we’ll be using), and boto (a Python-based way to access the Mechanical Turk API) with the following command: easy_install Flask-SQLAlchemy boto If you get a permissions error, try: sudo easy_install Flask-SQLAlchemy boto (which will prompt you for your password)
• To check out the most recent version of psiTurk type: git clone git://github.com/NYUCCL/psiTurk.git
• First, edit the config.txt.example file and save as config.txt (this is heavily commented and should be described better on the Wiki soon). Next, you can start the server using python app.py
• From there, we strongly recommend that you check out the API slides to learn a little more about how the system works.
• John also recommend the gunicorn which simply replaces the built in Flask/Python webserver with one that is a bit more robust.

MVPA

The second presentation was lead by Stephanie Chan and Reka Daniel-Weiner on MVPA. The MVPA workshop was be split into two main parts.

• First Stephanie and Reka discussed general issues of doing MVPA experiments, including some advice on experimental design and an outline of the analysis steps. You can get the slides here: pdf
• Next, they presented two different ways to actually do the analysis: Stephanie showed off the Matlab-based “Princeton MVPA toolbox” (http://code.google.com/p/princeton-mvpa-toolbox), and Reka presented the python-based PyMVPA toolbox (www.pymvpa.org).
• Instructions for installing the Princeton toolkit are linked above. The PyMVPA website provides instructions for all major operating systems: PyMVPA instructions
• The easiest way might be to download virtualbox (www.virtualbox.org) and run the NeuroDebian virtual machine on it (http://neuro.debian.net/vm.html) — then all the dependencies are automatically there after typing sudo apt-get install fsl ipython python-mvpa2 python-mvpa2-doc The rootpassword for the virtual machine is “neurodebian”.
• The tutorial data is available from here

While the benefits of Git are obvious in terms of backing up and sharing with others through services such as Github, the branching and tagging model used by Git can be uniquely suited to the sorts of organizational issues cognitive scientists often have to grapple with.

First, scientists need an easy way to archive code that has actually been run on human subject. Rather than making multiple copies of the code filling up you hard drive, this is facilitated by Git’s “tagging” feature (described more in the slides below).

Second, cognitive scientists often need to track multiple versions of the same codebase, as when multiple experiments in a paper implement the same task with different manipulations. Git’s branching feature (also described below) makes it easy to fix a bug in the experiment once, and then painlessly apply that bugfix to each version of the experiment.

A final feature of Git is the opportunities for collaboration. Git allows multiple people to be working on a project at the same time and can help merge those changes later. You can think of it as similar to Word’s “track changes” feature but it applies to any type of file, for example software code or papers written in LaTeX.

If this sounds interesting to you, you might want to take a look at my slides, which I’ve annotated and adapted into a web page, available here. Git has a slight learning curve if you are unfamiliar with version control software like subversion. However, I have provided specific commands that should hopefully make it easy for you to follow along. With a little disciplined use of Git it can make your scientific projects more organized, bug free, and shareable!

Our lab is already making extensive use of Git and Github. In fact, you can take a look at our public repositories here.

In this article

• Git for Scientists: A Tutorial.
• NYUCCL on Github

Is Internet-based data collection the future of cognitive science research? We used Amazon’s Mechanical Turk (AMT) to replicate a classic result in cognitive psychology which has primarily been established under traditional laboratory conditions (Shepard, Hovland, and Jenkins, 1961). In this post, we describe the various lengths we went to in order to get useful data from AMT and what we learned in the process. Overall, our results highlight the potential for using AMT in experimental research, but also raise a number of concerns and challenges. We invite comments, discussion, and shared experience below!

Note: Aspects of this post are now published in
Crump M.J.C., McDonnell, J.V., and Gureckis, T.M. (2013) Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE 8(3): e57410.

Table of Contents

Use the following links to jump around in the article:

• The Challenge of Collecting Behavioral Data
• Taking Experiments Online
• Exp 1: A Replication of Shepard, Hovland, Jenkins (1961)
• Exp 2: Do you get what you pay for? Exploring the effect of payment on performance
• Exp 3: An instructional manipulation check
• General Discussion
• Advice and Suggestions
• References
• Commentary

The Challenge of Collecting Behavioral Data

One challenging aspect of experimental psychology research is the constant struggle for data. Typically, we depend on university undergraduates who participate in studies in exchange for experience, course credit, or money. Research progress depends on the ebb and flow of the semester. As a result, it can take weeks, months, or even years to conduct a large behavioral study. This issue is even more salient for researchers at smaller universities.

One appealing solution is to collect behavioral data over the Internet. In theory, online experimentation would allow researchers to collect data internationally, enable access to a very large number of people who may be interested in participating in research studies, and can be fairly automated. However, the main obstacle to conducting Internet-based research is finding people who are willing to participate and compensating them. Sure, you can post a link on your webpage, but few people are likely to find it.

Recently, our lab (like a number of others) has explored the possibility of using Amazon’s Mechanical Turk (AMT) for behavioral data collection. AMT is a crowdsourcing or “artificial-artificial-intelligence” platform in which people submit simple (and usually brief) computer-based jobs to human workers online (see Box 2).

While Internet-based methods for collecting data have been around for a while, AMT is a potentially useful system for researchers since it handles both recruitment and payment in a fairly automatic way. There are a large number of people who use AMT making it a great way to advertise and distribute studies (the NYTimes reported over 100,000 active users in 2007).

Recently, there have been a number of excellent summaries and workshops about using AMT for research. Most notably, Winter Mason of the Stevens Institute of Technology has a Behavior Research Methods paper [pdf], which summarizes how to use the system and what it does (see also this excellent blog post and the references below).

Rather than focus on how to use AMT, this post focuses on the reliability of the data for experimental research in cognitive psychology. Before using this data in our experiments we wanted some sense of the quality of the data when compared to laboratory studies. Along those lines, we had a couple of questions going into this project:

• First, could we replicate classic findings in cognitive psychology concerning learning and memory processes with fidelity similar to that collected in the lab?
• Second, what unexpected issues come up in conducting learning, memory, and reasoning studies online?
• Third, how reliable is data on AMT as a function of the payment amount?

In this sense, our analysis is similar to a couple of recent papers (e.g., Paolacci, Chandler, & Ipeirotis, 2010 and Buhrmester, Kwang, and Gosling, 2011). However, these previous reports focus on survey data, the test-retest reliability of AMT, or on simple one-shot decision making tasks. In contrast, we were interested in using AMT to replicate and extend classic findings in experimental cognitive psychology which were originally established in the lab. Our emphasis was not just on qualitative replication, but in getting accurate data in a study that has been frequently replicated in the lab. In addition, we wanted to explore if it is possible to conduct publication-quality Internet experiments which unfold over many trials, require learning, sustained attention, and which may require between 20-60 minutes to complete.

Our initial experiments provide answers to many of these questions, and we felt our results might interest people thinking of using AMT for their own experiments.

Taking Experiments Online

In terms of taking cognitive experiments online, there are a number of unique challenges compared to simple ratings tasks or surveys.

First, such experiments usually take extended time (on the order of 1 hour) and uninterrupted concentration which may or may not be amenable to the AMT, which typically emphasizes short, one-shot human judgment tasks or surveys. In our initial test, we have tried to keep our tasks shorter than usual (roughly 15-30 minutes).

Second, compared to a simple HTML survey, the visual display in many cognitive experiments is typically dynamic (things may move around, there may be animations, people may respond by manipulating objects on the screen with the mouse, etc…). This raises a couple issues about how to control the screen display to ensure that people using many different types of computers (netbooks, iPads, regular desktops) view roughly the same thing. In addition, ensuring the experiment can even run without hassle in the Worker’s browser can be a technical challenge.

After initial testing and research, we decided that there is no general solution to these issues and it is up to the individual to balance ease-of-programming with the ability to support many diverse computer systems (this will be discussed more in an upcoming post). For example, we ran into difficulty in providing support for Internet Explorer in our task. It might have taken an extra two or three weeks to figure out all the idiosyncrasies of the various rendering engines and Javascript implementations in every browser. However, it turns out that most users on AMT do not use Internet Explorer so this decision has (hopefully) modest consequences.

Ultimately, there are many options available for designing web-based experiments that run in a browser and if you are savvy enough you can probably do 80% of what Matlab’s Psychophysics toolbox does. We found it fairly simple to design a dynamic display with Javascript that did not require extensive browser reloading between trials: All stimuli load at the beginning of the experiment and run as a web-app in the user’s browser sending us updated data at the end of each block.

Experiment 1: A replication of Shepard, Hovland, Jenkins (1961)

In our first experiment with AMT we attempted to replicate Shepard, Hovland, and Jenkins’ (1961) classic study on category learning. This is a highly replicated and influential study in cognitive psychology. In addition, it has many of the key features we want in our online experiments (the participant has to pay attention for 20-30 minutes and learn something over a number of trials).

Figure 1. Two examples of the Shepard problems. Eight stimuli (here square-like objects) are divided into two groups. We used the stimuli developed and normed by Love (2002).

As a summary, Shepard et al. (1961) had participants learn various categorical groupings of a set of eight geometric objects. Each of the six groupings varies in difficulty in a way related to the complexity of the “rule” needed to correctly partition the items. Differences in difficulty persist despite the fact that, in theory, people could memorize each item. This is often taken to imply people are forming more abstract, structured conceptions of the pattern (e.g., by forming a rule).

Figure 2. The abstract structure of the Shepard, Hovland, and Jenkins classification problems (taken from Love, Medin, & Gureckis, 2004). Each stimulus can be coded as a binary vector along the three stimulus dimensions. The problems differ in how the eight items are assigned to the two categories.

Two example problems are shown in Figure 1. The “Type I” problem requires participants to form a rule along a single dimension (‘if blue then category A, otherwise category B’). This problem is usually fairly easy to learn, while other problems are more difficult. For example, the rule in the “Type VI” problem is a complicated three-way XOR and is might be best learned by memorizing the category membership of each item. A full description of the abstract structure of the Shepard learning problems is shown in Figure 2.

In general, previous research has shown that the type I problem is learned more easily than is the type II problem. In turn, types III, IV, and V are learned more slowly than type II (within III-V, learning rates appear similar). Finally, type VI is typically the most difficult pattern to learn. The relative rate of learning for these six problems has provided an important constraint on theories of human concept and category learning. Most computational models of categorization must account for the relative difficulty of these problems in order to be viewed as a serious theoretical account. In addition, the quantitative (rather than qualitative) shape of the learning curves has been used to test and differentiate models (e.g., Love, Medin, & Gureckis, 2004). Our basic goal was to see if we could replicate this finding using participants recruited over the Internet.

Methods

Participants

Two hundred and thirty-four anonymous online participants volunteered (N=41, 38, 39, 39, 39, and 38 in types I-VI respectively), and each received $1.00 via AMT’s built-in payment system. In addition, 1 in 10 participants who completed the task were randomly selected for a bonus raffle of $10. This incentive was included to encourage people to finish the task even if they found it difficult. An additional fifty-six participants initiated the experiment electronically, but withdrew before the end for unknown reasons. The data from these participants was not further analyzed. Finally, seven individuals indicated they used pen and paper to solve the task in a post-experiment questionnaire and were excluded (although these participants still received payment). Participants electronically signed consent forms and were debriefed after the experiment. The study design was approved by the NYU Institutional Review Board.

We conducted our experiment between 1:30pm EST February 24th, 2012 and 6pm EST February 28th, 2012. Data collection was generally paused each evening at around 9pm EST and started again the following morning. A restriction was put in place that participants were located with the United States and had at 95% acceptance rate for hits. The purpose of this was to increase the probability that the participants were native English speakers who could fully understand the instructions and so we could keep data collection during relatively normal “working” hours. In addition, our experiment code checked the ‘Worker ID’ (a unique number assigned to each Worker account) and made sure that each unique account could only participate in the task once. People could evade this restriction if they had multiple Amazon accounts, but doing so would be a violation of Amazon’s Terms of Use policy.

Design

Each participant was randomly assigned to complete one of the six learning problems defined by Shepard et al. The stimuli were simple square objects similar to the ones shown in Figure 1. The stimuli we used were developed by Love (2002) who normed the constituent dimensions for roughly equal psychological salience. The mapping between the stimuli and the abstract structure shown in Figure 2 was randomly counterbalanced across participants.

Procedure

Our replication, although presented in AMT, remained procedurally similar to a highly cited replication of the Shepard et al. results by Nosofsky et al. (1994). On each trial of the task, one of the eight objects was presented in the middle of the browser window (see Figure 3). Participants indicated if the item belonged to category A or B by clicking the appropriate button. Feedback was then presented for 500ms which indicated if the participant was correct or incorrect¹

Figure 3. An example of the web-based interface. A single object was presented and people simply responded by indicating the correct category membership of the item. Corrective feedback was given following their response.

Trials were organized into block of 16 trials. In the rest period between blocks, participants were given information about their performance in the previous block and about how many more blocks remained. The experiment lasted until the participant responded correctly for two blocks in a row (32 trials) or until they completed 15 blocks. Participants were told that the experiment could last as long as 15 blocks, but that they could end early if they correctly learned the grouping quickly. Participants were asked not to use pen and paper.

After completing the task, participants filled out a brief questionnaire that asked if they used any external learning aids (e.g. pencil and paper), if they used any particular strategy, and how much they enjoyed the task and how difficult they thought it was.

Results

Figure 4 shows the probability of making a classification error as a function of training block for each of the six problems. If a participant reached the performance criterion (one block 100% correct) before the 15th block, we assumed they would continue to respond perfectly for all remaining blocks. Figure 4 is split in two panels. The data collected by Nosofsky and colleagues appears in the left panel and our AMT data appear in the right panel.

Figure 4. Probability of classification error as a function of training block. The left panel shows the learning curves estimated by Nosfosky et al. (1994) using 120 participants (40 per learning problem) who each performed two randomly selected problems. The right panel shows our AMT data with 242 participants, each who performed only one problem (between 38-41 per condition). We ended the experiment after 15 blocks, although Nosofsky et al. stopped after 25. Thus, the Nosofsky et al. data have been truncated to facilitate visual comparison.

There are a couple patterns of interest. First, people in the AMT experiment learn over trials and reduce the error rate. In addition, the type I problem was learned very quickly (within the first two or three blocks). In contrast, the error rate for the type II problem is somewhat higher (and more similar to problems III, IV, and V). As in previous reports, the type VI appears to be the most difficult. Thus, at a general level, our results are qualitatively in accord with previous findings.

At the same time, in all conditions besides type I, our participants performed significantly worse than Nosofsky et al.’s participants. For example, in all problems except for type VI, the probability of error in Nosofsky’s study fell below 0.1 by block 15. In contrast, our error rates asymptote near 0.2. One hypothesis is that participants on AMT generally learn more slowly, but this perspective is undermined somewhat by the fact that type I was learned at a similar rate to Nosofsky (the probability of error drops below 0.1 by the second block of trials).

This rather slower learning rate for the more complex problems is also reflected in Figure 5 (left panel) which compares the average number of blocks taken to reach criterion both participants in our data and for Nosofsky et al. In almost every problem, participants on AMT took nearly double the number of blocks compared to Nosofsky et al.’s laboratory study. Closer inspect of the data showed that this was due to a rather large proportion of participant who never mastered the problems at all (taking all 15 blocks).

Figure 5. The left panel shows the average number of blocks it took participants to reach criterion (2 blocks of 16 trials in a row with no mistakes) in each problem. The yellow bars in the right panel shows the estimated average number of blocks to criterion reported by Nosofsky et al. (1994). The right panel shows the proportion of participants in each condition reaching the learning criterion. Nosofsky et al. (1994) did not report their data in this way.

Figure 5 (right panel) shows the proportion of subjects reaching criterion before the end of the task. Roughly half the participants were able to master the problem by the end in problems II-VI. However, this view of the data does suggest that type II was at least marginally easier than problems III-V.

Figure 6. The number of drop-outs for each of the six problems. Dropouts are defined as people who started the experiment but didn't finish (e.g., closed the browser window).

Interestingly, the difficulty of the task (according to Shepard et al. and Nosofsky et al.) didn’t have a strong impact on people deciding to drop out of the task. To assess this we counted the number of participants who started the experiment but didn’t successfully finish as a function of condition (see Figure 6). As you can see, the drop-out rate does not seem to be systematically related to the problem difficulty (e.g., the smallest number of dropouts was in the type III problem was apparently somewhat difficult for participants).

It is also worth noting that we didn’t attempt any post-hoc “clean up” of the data (e.g., excluding people who took a long time or who pressed the same key for many trials in a row). While such exclusion may be warranted in certain cases, we didn’t have clear a priori hypotheses about which kinds of exclusions would be appropriate for this data. However, given the large percentage of subjects who failed to master the problems within 15 blocks, it is unlikely that there is a simple exclusion criterion that would make our data align well with the Nosofsky et al. replication (without directly excluding people who didn’t learn).

Experiment 2: Do you get what you pay for? Exploring the effect of payment on performance.

The above results were mixed. On one level it was definitely encouraging to collect so much data in such a short time frame. In addition, participants did tend to learn (e.g., in the Type I problem and as reflected in the overall error rates). However, at least when compared to Nosofsky et al. (1994), learning performance in our replication was considerable lower.

One possibility is that if we incentivized participants to perform well, we could get better data. Put another way, is the quality of AMT data basically as good as you are willing to pay? As noted by Gosling et al. (in press) some AMT workers seem to participate mainly for personal enjoyment, and payment isn’t such a important issue for these individuals. For example, in their report, they found that a large number of Workers would complete a survey for $.01 (the minimum possible payment).

However, this does not apply universally. Anecdotally, we attempted to run the Shepard et al. study reported above again but only offered $.25 as payment (and no lottery or bonus). In that case we recruited only 1 subject in 12 hours (2 others dropped out after the first block of the task). Thus, workers are influenced to participate by the magnitude of payment and their estimation of the difficulty or length of the task. However, this sensitivity to the possible payment might also influence task performance in theoretically significant ways.

In a second study, we tried to systematically explore how our replication results might depend on how much money the AMT workers are offered. This issue is rarely examined systematically in the laboratory but could have important implications in online data where participants decision to participate may be a more purely economic decision.

Specifically, we repeated the above study with two different incentive structures. We felt our initial payment scheme described above was roughly in line with what we would pay a laboratory subject for a short 15-20 minute task ($1.50 on average). Thus, we created two additional conditions, a “low incentive” group that was paid $0.75 and not offered a bonus. A second “high incentive” group was offered a guaranteed $2 and a bonus of up to $2.50 based on task performance.

Rather than test all six Shepard et al. problem set we focused this analysis on the II and IV problems. By comparing the results of this replication with our previous set of data we hoped we could obtain information about the relative effects of payment on the relationship between our online replication and related laboratory studies.

In addition, we collected demographic information about participants in this study.

Methods

Participants

Eighty-two anonymous online participants volunteered and were evenly divided between either a “low incentive” or “high incentive” condition. Within each condition, participants were randomly assigned to either the type II or type IV problems (N = 20 or 21 in each condition for both incentive conditions). In the “low incentive” condition each participant received $0.75 via AMT’s built-in payment system. There was no bonus or lottery offered for these participants. In the “high incentive” condition, participants in were paid a base amount of $2 for completing the experiment and a bonus of up to $2.50. The bonus was calculated as follows: at the end of the experiment, 10 random trials were selected from the participant’s data file and each trial where the participant provided a correct response increased the bonus by $.25. If the participant reached criterion (2 blocks with 100% correct responses) we coded all remaining trials as correct. This placed a relatively stronger financial incentive on quickly mastering the problem compared to either the “low incentive” condition or the previous experiment.

An additional twenty participants initiated the experiment electronically, but withdrew before the end for unknown reasons or self-reported using pen and paper to complete the task. As before, a restriction was put in place that participants were located with the United States and had at 95% acceptance rate for previous “hits”.

We collected data for the low-incentive condition during a 25 hour period beginning March 9th, 2012 at 5pm EST and ending March 10th at 6pm EST. Data collection was stopped at 9pm EST each evening and began again after 10am EST. We collected data for the high-incentive condition during a 2 hour period beginning March 20th, 2012 at 3:30 EST and ending 5:30 EST.

Design and Procedure

The design was mostly identical to the previous study except participants only completed either the type II or type IV problem. The procedure was mostly identical to before, the only difference was the incentive (either high or low).

Results

Figure 7 compares the learning curves for both the type II and type IV problems across three incentive conditions (the “medium” data is the same as above). As you can see, the incentive structure of the task had very little impact on overall learning rates in the task and does not fundamentally change the impression that the type II and type IV problems were learned at a roughly similar rate. This result aligns well with Mason and Watts (2009) who report that the magnitude of payment doesn’t have a strong effect on the quality of data obtained from online, crowd-sourced systems.

Figure 7. The learning curves for Shepard et al. problems II and IV based on task incentives. The incentive structure had little impact on participants

The effect of incentives on drop outs and recruitment. Overall, the incentive structure in the task had little impact on learning performance. However, it did strongly influence the rate of signups (40 subjects were collected in 2 hours in the high incentive condition while it took roughly two days to collect the same amount of data in the low incentive condition). In addition, it strongly influenced the dropout rate. In the high incentive condition, only 5 participants started the task but did not finish (2 in type II and 3 in type IV), giving a dropout rate overall of 11%. In contrast, 13 participants in the low incentive condition started but did not finish the task (six in type II and seven in type IV), for an overall dropout rate of ~25%. Again, this result is largely consonant with Mason and Watts (2009).

Experiment 3: An instructional manipulation check

Our results so far are interesting, but also suggest caution in using AMT data in cognitive science research. Despite some faint hints of the classic learning pattern in our data, there were fairly large discrepancies between our study and laboratory collected data. This mostly manifested in significantly worse learning for the conditions requiring “complex” cognition (problems II-VI, relative to the simple one-dimensional rule used in problem I). One concern is that the variable testing environment online contributes to distraction or lack of participant motivation that might negatively impacts performance in more challenging cognitive tasks. This would tend to reduce the utility of systems like AMT for research on these topics.

However, rather than give up, we doubled down in our efforts. First, we made some changes to our experiment code to be more in line with Nosofsky et al’s original replication. In particular, we replaced the stimuli developed by Love (2002) with the simple geometric figures used by Nosofsky et al. and Shepard. Pilot data suggested that the stimulus differences were not the main factor influencing performance but to ensure more comparable results we thought it would be prudent to minimize all differences.

Second, we became concerned that some participants may not have completely understood the instructions (some responses to the post-experiment questionnaire indicated that people believed the rule was changing from one block to the next). It seemed very likely that a failure to fully understand the instructions would negatively impact performance, perhaps differentially on the more difficult problems.

To address this issue, we incorporated an instructional manipulation check which has been shown to reduce noise in behavioral experiments (Oppenheimer, Meyvis, and Davidenko, 2009). This (rather straight-forward) technique requires the participant to answer non-trivial comprehension questions about the instructions of the experiment before participating. While Oppenheimer et al. introduced somewhat insidious “gotcha” questions into their instructions, we simply presented participants with a questionnaire at the end of the instruction phase which tested knowledge of the basic task and study goals. Correct answers to the questionnaire required a complete comprehension of the goals of the experiment and addressed possible misconceptions (e.g., “Will the rule change on each block?”, “Is it possible to get 100% correct?”, “Should you use pen and paper to solve the task?”, etc…). If a participant incorrectly answered any of the questions, they were asked politely to read the instruction again. This process repeated in a loop until the participant was able to answer all the comprehension question correctly.

Methods

Participants

Two-hundred anonymous online participants volunteered and were randomly assigned to either the Type I, II, IV, or VI problems (N = 50 in each). Participants were offered $1 to complete the task along with a one in ten chance of winning a $10 bonus (only available if they completed the task). This matches the “medium incentive” condition used in Experiment 1.
An additional 33 participants initiated the experiment electronically, but withdrew before the end for unknown reasons or self-reported using pen and paper to complete the task. As before, a restriction was put in place that participants were located with the United States and had at 95% acceptance rate for previous “hits”. We collected data beginning March 29th, 2012 at 11:30am EST and ending April 2nd at 5pm EST. Data collection was stopped around 9pm EST each evening and began again after 10am EST.

Design and Procedure

The design was identical to the previous study except participants only completed either the Type I, II, IV, VI problem. The only major change was to the stimuli (made to match Nosofsky et al, 1994) and the instructions (detailed above). The procedure was identical to before.

Results

Figure 8 (left panel) compares the learning curves for Nosofsky et al. (1994) and Experiment 3. The most striking pattern is the closer correspondance between our AMT data and the laboratory-collected data for problems I and IV. These data probably fall within the acceptable margin of error across independent replications of the laboratory study. As an illustration, the right panel compares the Nosofsky et al. data to a independent laboratory-based replication by Lewandowsky (2011)². Given the intrinsic variability across replications, this suggests the AMT data do a fairly good job of replicating the laboratory-based results. In contrast, the type VI problem appears a little more difficult for participants on AMT compared to in the lab. However, at least compared to our results in Experiment 1, the relative ordering of the problems is much more pronounced (i.e., I is easier than IV which is easier than VI).

Figure 8. Left panel: A comparison of the learning curves for Experiment 3 and Nosofsky et al. (1994). The Nosofsky data have been truncated to show 10 learning blocks. Relative to Figure 1, the results look much more in-line with the laboratory-based study. For comparison, the left panel shows the alignment between Nosofsky et al. (1994) and a more recent laboratory replication by Lewandowsky (2011). Note that Lewandowsky (2011) also found a relatively small difference between the type II and type IV problems.

Despite generally increased alignment between the laboratory data and AMT data, anomalies remain. In particular, the type II problem seems systematically more difficult for participants in our online sample than in Nosofsky et al.’s laboratory study. This is clear in Figure 10 which breaks the learning curves up by problem type. The colored solid lines are Nosofsky et al. (1994), the dashed colored lines are our Experiment 3 results, and the solid black line is our Exp. 1 results. On the other hand, it is clear that the “instructional check” manipulation greatly improved performance in all condition (except perhaps type I which was already near floor).

Figure 9. A direct comparison of the learning curves from Nosofsky et al. (1994), Experiment 1, and Experiment 3. As in Figure 9, the data have been truncated to 10 blocks. The results show a relatively close correspondance between the AMT data in Experiment 3 and the performance of laboratory-tested individuals for problems I, IV, and VI (although VI is a little more difficult for Turk participants). In almost all conditions, the instructional manipulation check in Experiment 3 greatly improved learning (much more than did the incentive manipulation in Experiment 2.

The finding that type II is learned roughly at the same rate as type IV is interesting. However, other measures of learning suggested at least a marginal type II advantage. For example, 100% of participants in the type I problem reached the learning criterion within the 10 training blocks (2 blocks in a row with 100% correct responses). In comparison, 73.1% reached criterion in the type II problem. However, only 56.4% reach criterion in the type IV problem and 44.8% reach criterion in the type VI problem. Interestingly, our finding of similar learning curves for the type II and IV problems has some precedent in the laboratory literature. For example, as visible in Figure 8, Lewandowksy (2011) found that the type II problem was learned at roughly the same rate that the type IV problem. A similar result was reported by Love (2002) who found only a marginal type II advantage compared to the type IV problem in a related design. We’re in the process conducting a couple followups to this result which might shed some light on this issue.

General Discussion

Overall, our experiments with AMT seem promising, but also raise some interesting issues.

First, it was amazing how much data we could collect in a short period of time. Performing a full-sized replication of the Nosofsky et al. (1994) data set in under 96 hours is revolutionary. This alone speaks volumes about the potential of services like AMT for accelerating behavioral research.

Second, it is notable that participants did learn in all conditions (error rate dropped from the beginning to the end of the study in all conditions). This fact was not necessarily a given since people could have chosen to respond randomly. Manual inspection of our data suggests this almost never happened.

Third, many participants were willing to take part in the 15-30 minute study even when offered $.75 in the low incentive condition. Given that this is about 2-3 times longer than typical HITs on the system suggest there is a reasonable market for recruiting human participants. In our high incentive condition, we could run as many as 40 participants in two hours.

Taken together, the results are extremely encouraging. In less than 96 hours, we collected a 272 person cognitive psychology experiment over the internet. Despite a presumably heterogeneous testing environment and population pool, we seem to have gotten sensible data that qualitatively replicates previously reported results from the lab.

Finally, we replicated the key finding of Shepard et al. and Nosofsky et al. (type I was easier than types III-V which are easier than type VI). At the same time our data was a little less clear than the previously published laboratory collected studies. In general, type II seemed slightly more difficult than previously reported (at least in our learning curve analysis). At this point we are not sure what to make of this difference, except to point out that a couple recent laboratory studies report a similar pattern (e.g., Love, 2002; Lewandowsky, 2011). In addition, online participants generally learned more slowly (this was especially true in Exp 1 and 2 but also showed up in the type VI condition in Exp. 3). It may be that the slower learning relates to the more diverse participant sample than is typical in laboratory studies (e.g., we did find a slightly negative correlation between performance on the type II problem and self-reported age).

However, more than anything, we found that building in checks for understanding the instructions is critical for ensuring high quality data. After incorporating those changes, our data began looking more like a publication-quality replication study. Overall, a pretty worthwhile exploration.

A quick survey of the cognitive science literature suggests that Internet-based studies have not yet made it fully into mainstream cognitive journals. There are numerous Cognitive Science conference proceedings papers which use Turk data, and a few social psychology studies which use Turk to collect surveys (please use the comments if there are others we are unaware of!). However, considerably fewer traditional experimental psychology papers have published Internet data as a primary source of subject recruitment. Based on the above, it seems that reviewers and editors might consider accepting behavioral experiments done on AMT as a valid methodology (applying as much scrutiny as they would apply to any behavioral paradigm). Even for extended experiments requiring problem solving and learning, the data seem mostly in line with a laboratory results.

Advice and Suggestions

To conclude, we’d like to offer a bit of practical advice based on our experience.

First, we echo the point made by Mason and Suri (in press) that researchers pay AMT users something close to minimum wage in the United States or at least close to what you offer someone to perform the task in the lab. While our above analysis suggests that low pay doesn’t necessarily effect the quality of the data, we have found that we can recruit participant faster and have fewer dropouts by making the study financially appealing. In addition, it seems ethical to attempt to roughly equate payment across the lab and “Internet lab” (recognizing that a computer study requires a bit less than traveling to a lab to take part in the study). Many companies offer simple HITs on AMT for as little as $.10, but such rates are somewhat out of line with what subjects in the lab are offered.

Second, experiments that are at least somewhat fun and engaging are likely to be better received. If you have people making 5000 discrimination judgments for simple lines or sine-wave gratings it seems less likely you will get highly useful data. One way to view it is that you are basically competing against all the other interesting things to do on the Internet (YouTube, etc…). On the other hand, in our “comment box” at the end of our experiment, many of the participants said they found the rule-discovery task to be fun and interesting (but to be fair, others hated it!).

We considered various ways to exclude “suspicious” or “odd” behavior (e.g., pressing the same key many times in a row, long response times) but ultimately didn’t report those analyses above. The problem is that our exclusion criteria were entirely arbitrary. Generally, we do not advocate excluding participants except under the most extremely obvious situations of abuse (pushing the same button the entire time). However, restrictions should be decided before data collection and clearly reported in papers. Also the time of day and date of data collection may be important as the AMT population may evolve over time.

Most importantly, we found that testing participant’s comprehension of the instructions was critical. Prior to including such checks our data were much noisier. In fact, the instruction check had a considerably more robust effect on the quality of our data than did increasing the payment. In retrospect this point is intuitive, but it was a lesson worth having sooner rather than later.

Finally, it is important to monitor and record the rate at which people begin your experiment but do not finish. This is typically not a problem in laboratory studies since the social pressure of getting up a walking out of the lab is much higher than it is online. However, dropout rates can interact in complex ways with dependent measures such as accuracy (low performing individuals may be more likely to drop out online). We recommend that, perhaps unlike a typical laboratory study, all Internet experiments report dropout rates as a function of condition.

The bottom line? AMT certainly seems promising for experimental cognitive science research. Our investigations suggest that the data quality is reasonably high and compares well to laboratory studies. Hopefully, the quality of the data will remain high as additional researchers start to utilize this resource. If we (scientists) respect the participants and contribute to a positive experience on AMT it could turn into an invaluable tool for accelerating empirical research.

References

Gosling, S.D., Vazire, S., Srivastava, S., and John, O.P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist, 59, 2, 93-104. [pdf]

Lewandowsky, S. (2011). Working memory capacity and categorization: Individual differences and modeling. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 720-738. [PDF]

Love (2002). Comparing supervised and unsupervised category learning Psychonomic Bulletin & Review, 9(4), 829-835.

Love, B.C., Medin, D.L., and Gureckis, T.M. (2004) SUSTAIN: A Network Model of Category Learning. Psychological Review, 11, 309-332.

Mason, W. and Suri, S. (in press). A Guide to Behavioral Experiments on Mechanical Turk. Behavior Research Methods [pdf]

Mason, W. and Watts, D. (2009). Financial incentives and the “performance of crowds.” HCOMP ’09: Proceedings of the ACM SIGKDD Workshop on Human Computation, 77–85. [PDF]

Nosofsky, R.M., Gluck, M.A., Palmeri, T.J., McKinley, S.C., and Glauthier, P. (1994). Comparing models of rule-based classification learning: A replication and extension of Shepard, Hovland, and Jenkins (1961) Memory & Cognition, 22(3), 352-369. [PDF]

Oppenheimer, D.M., Meyvis, T., and Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867-872. [PDF]

Paolacci, G., Chandler, J., and Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419. [pdf]

Shepard, R.N., Hovland, C.I., and Jenkins H.M. (1961). Learning and memorization of classifications. Psychological Monographs: General and Applied, 75(13), 1-41.

1. Our experiment code directly modified the current browser window and sent data to the server during rest sessions using AJAX. This meant there was no reloading or network access between individual learning trials. While our timing estimates of the display are approximate (depending on the user’s browser, speed of their computer, etc…) they are substantially more accurate than if network access was required on every trial.
2. These data were estimated from the Figures in Lewandowsky’s paper

A forest for the trees is a stimulus set designed by Todd Gureckis and Nathaniel Blanco for use in a dimensional categorization task with preschool aged children (a ongoing collaboration with Marjorie Rhodes).

The stimuli are illustrations of trees that vary along two dimensions – the “bushiness” of the leaves:

and the color of the leaves:

The stimuli are relatively rich and realistic and made for a compelling cover-story/game for young children. The basic task was to categorize of the trees into two groups. The children had to learn the basis for the categorical grouping (either business, color, or some combination of both).

Some of the items looks somewhat similar when viewed in isolation, but when arranged in a row the structure is clear. The stimuli can be used as is, or a subset along each dimension could be used to maximize the between stimulus differences.

Here is the full set:

It was inspired in equal parts by Jared Tarbell’s computational tree garden and by New York’s Adirondack fall.

Illustrator and .png versions of the graphics are provided in the following zip file:

All images copyright to Todd Gureckis and Nathaniel Blanco! Feel free to use them in your projects if they end up useful!

Farrell, S. and Lewandowsky, S. (2010). “Computational models as aids to better reasoning in psychology” Current directions in psychological science, 19(5), 329-335. [pdf]

Abstract: “Scientists can reason about natural systems, including the mind and brain, in many ways, with each form of reasoning being associated with its own set of limitations. The limitations on human reasoning imply that the process of reasoning about theories and communicating those theories will be error prone; we must therefore be concerned about the reproducibility of theories whose very nature is shaped by constraints on human reasoning. The problem of reproducibility can be alleviated by computational modeling, which maximizes correspondence between the actual behavior of a posited system and its behavior inferred through reasoning and increases the fidelity of communication of our theories to others.”

Choice quote: “…computational modeling helps ensure reproducibility in scientific thinking. By implementing a model as a computer program or a set of equations, another researcher can take our model and exactly reproduce our predictions.”

Best bit from the article: “The reason Target can snoop on our shopping habits is that, over the past two decades, the science of habit formation has become a major field of research in neurology and psychology departments at hundreds of major medical centers and universities, as well as inside extremely well financed corporate labs. “It’s like an arms race to hire statisticians nowadays,” said Andreas Weigend, the former chief scientist at Amazon.com. “Mathematicians are suddenly sexy.””

Clearly this is something my undergraduates need to know!

Finally, open the Terminal app (/Applications/Utilities/Terminal.app) and change to that directory (the easiest way to do this it to type “cd” then drag the folder into the Terminal window, the press enter. Then type “chmod u+x launchipython.command.”

After this, if you double click the file, your default browser should pop open with a view of the current folder’s notebooks.

An example of a in-progress Mathematica analysis!

For those who haven’t used it, the Mathematica notebook allows you to combine text, code, and figures/plots in a single multi-media document. This is very helpful for building up and testing complex bits of code and for exploratory data analysis. Rather than cutting and pasting bits of code from a text editor into an interactive interpreter (as in Matlab, Python, or R) the notebook allows code and graphics to coexist in line with one another. It turns out it is very helpful to keep the graphics from one analysis tied to the code that generated it (along with plain text describing it). For example, if I write some code for a data analysis I can have the plot of this data appear directly below the code itself in the notebook. Later, when I’m going back through the analysis (perhaps weeks or even months later) I can more easily make sense of what goes with what. It’s basically just a more logical way to relate the _outputs_ of a computation to the code that generated it (i.e., “literate programming“).

Anyway, as great as Mathematica is, there are a couple of important draw backs.

First is that Mathematica isn’t free (as in beer or as in speech). This isn’t such a big deal for us (NYU has a university-wide academic license which keeps the cost lower). However, it is hard to involve undergraduates in the research process since it might be too costly to buy licenses for all of them. I’m also reluctant to ask students in a class I’m teaching to shell out for a license (even though Wolfram Research has made this easier lately via their per-semester licensing deals). In addition, I worry that my psych undergrad or grad students are less likely to encounter Mathematica again (whereas they are somewhat more likely to run into R, SPSS, Matlab, or Python).

A second disadvantage to Mathematica is that it isn’t always as fast as something like Matlab or Python. I’m not entirely sure why that is (some matrix computations are optimized), but it likely has to do with the very powerful symbolic computation tools that Mathematica provides. In many cases, this can make programming much easier. However, once a complex model or simulation is set up, we often find it is more effective to translate it into Python which runs runs fast enough for most of our work.

Finally, the Mathematica programming language is pretty old. It doesn’t have very clean object-oriented design patterns and the syntax can be a bit obtuse. For example, compare these two statements which do the same thing, one in Python and one in Mathematica:

def myfunction():
    for i in range(10):
        if i > 5:
            print "This number is greater than 5 ", i
        else:
            print "This number is less than 5 ", i

myfunction[] := Module[
    {i},
    For[i=1, i<=10, i++,
        If[i > 5, 
           Print["This number is greater than 5 ", i],
           Print["This number is less than 5 ", i];
         ];
     ];
];

The bottom line is that Python’s language is cleaner, it is more contemporary, it runs faster, the number of available libraries is immense (at least equal to, if not exceeding, the functionality in Mathematica), and it is free/open source. We use Python internally for all our experiments (check out our simple API for developing psychology experiments, PyPsyExp).

Given all this, wouldn’t it be great if Python had a notebook interface?

Well, recently, the possibility of leveraging some of the benefits of Mathematica’s “computational notebook” framework in Python has emerged (thanks to Jay Martin for telling me about this!). In particular, iPython (an “enhanced” python shell) has added a web-based notebook framework. I’ve been playing with the bleeding edge version in Github lately and I’m impressed (thus, this blog post!).

The basic idea of the system is that you launch a small webserver running on your computer (using the command ipython notebook –pylab inline). Then, you point your favorite browser (I’ve found things work very well in Chrome) at a particular local URL it prints out (e.g., http://127.0.0.1:8888). From there, the web application serves up an interactive notebook instantiated as a web page. It might not seems like a web-based interface would be really useful, but advances in AJAX have enabled fully complex, dynamic applications that run in your browser (think Facebook or Google Docs).

The current notebook format feels quite a bit like Mathematica’s notebook interface. There is the concept of a “cell” which links a bit of executable code and the resulting output. Cells can also hold text, Markup, LaTeX, or other types of text. In addition, a system is worked out for showing graphics from pylab/matplotlib, perhaps the most ubiquitous data plotting library for Python.

However, at the current stage, Mathematica’s notebook format is still much more refined. For example, you can’t change the color of cells, can’t collapse/hide chunks of cells/code at a time, can’t execute multiple cells at one (or groups of cells) in iPython notebooks. In addition, since Mathematica has a much more robust graphics system it is easier to export the resulting graphics files for “clean up” in Illustrator. Since iPython is a web-based app all graphics are converted into something like .png files for display which are harder to subsequently edit (although, of course, you can use matplotlib to write to a file on your local disc). Despite these limitations, development of iPython Notebook seems active (at least by the discussion on Github), and I’m sure many of these things will be addressed as time goes on.

p.s. This is a great link for getting it set up on Mac OS X Lion: http://minrk.posterous.com/install-ipython-qtconsolenotebook-on-osx-lion.

UPDATE: See also this page which I will be updating for my course with install instructions for various operating systems. Also, Fernando Perez (original author of iPython) shared this link about the history of the project and this link about scientific python.

About Us
We are interested in unlocking the secrets of human cognition (i.e., reasoning, thinking, learning, decision, and memory processes) through a combination of behavioral experiments, formal modeling, and cognitive neuroscience methods (e.g., brain imaging). As a grad student in the lab, your job is to lead grounding breaking studies while being trained by some of the most respected psychologist and neuroscientists in the world (i.e., the general faculty at NYU). Collaborations are frequent between different labs and between students/advisors within the program. Research ideas quickly evolve from idea to implementation, and students “own” the work they contribute to. Graduate students in my lab are encouraged to be independent scholars and lead via their own interests and ideas even early in their graduate career (at least earlier than most PhD programs). I’m a “hands-on” advisor. I’m typically in the trenches debugging code with you and hashing out ideas. I’m invested in getting your work published and sharing it with the world and want to help you get there.

Who we are looking for
We are particularly interested in students with a strong quantitative background (e.g., Cognitive Science degree, computer science courses, engineering, etc…), those who have programming experience, and who are interested in a smooth blend of psychology, computational modeling, and even a little cognitive neuroscience. The work we do is often of a technical nature. A slogan in the lab is “If you can’t build it, you don’t understand it.” and that philosophy is why we place a strong emphasis on computational modeling. We prefer arguing about a equation or model than about a vague, verbal theory. An intuition for human behavior is key. Also, an eye for aesthetics and design is also a big plus (“Good design is a lot like clear thinking made visual” – Tufte). More than anything, we want to work with fun, ambitious people who are driven by curiosity, creativity, and rigorous scientific thinking.

If this sounds like a description of yourself (and sounds like something you’d like to spend a minimum of 5 years pursuing) please get in touch. You will have to apply through the standard NYU graduate admissions process, but there are places in the application form where you can indicate who you are most interested in working with. If you list me, be assured I will look closely at your application. Hope to hear from you!

From the article: “This is not to say there is anything quantum going on in the brain, only that “quantum” mathematics really isn’t owned by physics at all, and turns out to be better than classical mathematics in capturing the fuzzy and flexible ways that humans use ideas.”

Searle is wrong, btw.

The 33rd annual meeting of the Cognitive Science Society is kicking off next Wednesday. The lab is going to be well represented this year with a couple of talks and posters. Here’s a preview of what we’ll be presenting. Be sure to drop by the talks and posters and give us a hard time (or buy us a beer if it looks like other people are giving us a hard time!). Hope to see you there!

Thurs, July 
21

6:45pm -


 Posters


 (
Track 
A
, 
Poster 
Room)
Poster #1387
 – 
Does 
Category 
Labeling 
Lead 
to 
Forgetting?
Nathaniel Blanco, and Todd Gureckis

A closer look at claims that verbally labeling an object impairs memory for it. Sneak peak: probably not.

Fri, July 
22

6:45pm -


 Posters


 (
Track 
A
, 
Poster 
Room)

Poster #1298 
- 
Modeling 
information
 sampling
 over
 the 
course
 of
 learning
Doug 
Markant, and Todd
 Gureckis

An investigation into how people use their uncertainty to guide sampling decisions during a self-directed learning task and how their decisions change during the course of learning.

6:45pm -


 Posters


 (
Track 
A
, 
Poster 
Room)
Poster #1059 – 
Learning 
categories 
from 
an 
intermittent 
teacher
John 
V 
McDonnell, and Todd
 
Gureckis

A study of how, and under what circumstances, information from labeled and unlabeled items is integrated during category learning tasks where both types exemplars are provided.

Sat, July 23

1:40pm – 


Symposium:
 Grow 
your
 own 
representations:
 Computational
constructivism


 (
Track
 B
,
 Plaza
)
featuring Joseph 
Austerweil,
 Thomas
 Griffiths,
 Todd
 Gureckis,
 Robert 
Goldstone, Kevin
 Canini, 
Matt
 Jones

A invited symposium on “constructivist” approaches to human learning (i.e., ones in which new representations are built to interpret the world). Todd is giving a “endnote” talk discussing the various modeling approaches to this question and what they tell us about the mind and brain.

3:10pm – 


Timing 
and 
decision 
making 


(
Track 
F
, 
Georgian 
Room
)
Talk 603
 – 
Don’t 
Stop 
‘Til 
You 
Get 
Enough: 
Adaptive 
Information 
Sampling 
in 
a Visuomotor
 Estimation 
Task
Mordechai 
Juni, 
Todd 
Gureckis, 
Laurence 
Maloney

How much information is enough to make a decision? This talk presents a study of how people sample information in a visuomotor estimation task where each cue requested reduces possible reward.   Todd will be presenting.

Disclaimer: I tend to think modeling paradigms are a waste of time although they tend to generate lots of “heat” in the psychological literature. The goal of a model is to tell us something interesting psychologically or neurally. Everything else is just noise.

Last week Sen. Tom Coburn (R-OK) released a report attacking the National Science Foundation for waste, fraud, and mismanagement. Included in his report is a long list of research projects which he deemed frivolous and a waste of tax-payer money. I was quite surprised to learn that a recent paper I wrote with Rob Goldstone about social influence and individual decision making was one of the projects called out in the report. I have since been contacted by a number of people requesting information about NSF’s involvement in the research. In response, I worked up a short frequently asked questions (FAQ) which I thought I’d share here.

1. Coburn’s report alleges that “Armed with a $1 million grant from NSF, researchers at Indian (sic) University-Bloomington and New York University analyzed baby names to determine trends in parents’ naming decisions.” Did Indiana and NYU actually receive $1 million from NSF to study baby naming?

No.

First, NYU did not receive any funds from NSF related to this project as I was not a PI or Co-PI on the grant in question.

Second, the NSF grant in question was awarded to a co-author of the paper (Robert Goldstone from Indiana University). The footnote included in Coburn’s report references NSF award #0910218. The title of this grant is “Transfer of perceptually grounded principles” and originated from NSF’s Division of Research on Learning in Formal and Informal Settings. Thus, the grant was not to study names as claimed in the report. This fact alone largely undermines the criticism.

More troubling is that, actually, this particular grant was never even acknowledged in connection with the baby names research. Instead, we acknowledged NSF Award #0527920, a grant awarded to Rob Goldstone concerning “the use of interactive computer simulations to teach scientific concepts governing complex adaptive systems.” (Total award amount for this grant was $196,086 over 4 years, a far cry from $1 million). Again, as is clear from the abstract and title of this award, this grant was not to specifically fund baby naming research.

In contrast to the claim in the report, nobody received $1 million from NSF to specifically fund the research reported in Gureckis & Goldstone (2009).

Rob credited NSF in our paper because the paper deals with complex systems: namely, cultural transmission systems. Since the aforementioned grant was designed to improve the way we teach students about such complex scientific phenomena, he included a mention of his support in the paper. This paper counts as a “synergistic” activity related to the primary focus of Goldstone’s awarded project. Goldstone has also published many highly cited peer-reviewed papers on student learning which credit his NSF support (a list appears in the above link). Note that NSF requires all products of funded research to acknowledge said funding even if the ideas are only partially related to the original award. The paper also acknowledged funding from NIH/NIMH (a Mathematical Modeling Training Grant awarded to Indiana University which paid for my post-doc when I was at Indiana University), and the Department of Education. Other aspects of the writing of the paper were supported by private funds given to me by NYU when I first took my job here.

Thus, the claim in the report is objectively false, sensationalized, but also suggests a troubling lack of understanding about how scientific research is funded (confusing the multiple products of that research for the grant itself). The bottom line is that nobody received $1 million from NSF to study naming behavior (although, as described below, there is no reason why research on this topic shouldn’t be funded). The actual cost to the tax payer through NSF to produce this particular research report had to be less than $300 (and was quite possibly $0) but NSF was credited out of an abundance of caution (and thanks) for their support.

2. Did Coburn or his staff even look at the scientific paper in question?

Rather than examine the actual peer-reviewed research paper (Gureckis, T.M. and Goldstone, R.L. (2009) How You Named Your Child: Understanding The Relationship Between Individual Decision Making and Collective Outcomes. TopiCS in Cognitive Science, 1 (4), 651-674. available for download here), the Coburn report exclusively references a USA Today article written by an individual not involved in the original research. This news story is not an authoritative source on the contributions of our scientific paper. It would be like referring people to a left-wing blog to describe Coburn’s stance on political issues instead of letting them look at his own voting record or website.

The Coburn report exclusively references a USA Today article written by an individual not involved in the original research.

Had those developing this report actually looked at the research paper they were criticizing, they would know that we were not specifically interested in baby names except in so far as they offer a unique opportunity for studying such the impact of social influence on decision making. We all know that iPhones are popular but the underlying reasons for this cultural success is distorted by the role that advertising budgets and existing computer technologies play in determining which ideas win out and which die off in the consumer marketplace. In contrast, the popularity of names is more organically determined by processes of social influence (there is no company out there trying to convince you to name you child something in particular). Baby names thus represent an important and relatively “pure” empirical test of theories of cultural transmission and social influence in large groups.

The Coburn report makes it seem as though this research spent money to determine the frequency and popularity of names. Fortunately, this data was provided for free by the Social Security Administration which has recorded and published the most popular baby names in the United States since the 1880s (freely available here: http://www.ssa.gov/oact/babynames/). Many of the popular websites that analyze naming trends rely on the same data source. Any NSF funds used toward this effort paid exclusively for the statistical/mathematical analysis of this data. In fact, in the context of a discussion about government waste, this is a great example of government efficiency since data collected for one purpose (issuing social security cards), which would have been very expensive to collect otherwise, turns out to be very useful to NSF and NIH supported peer-reviewed science.

Note that many researchers agree that this data is unique for studying the interactions of individual decision making and social behavior. Similar analyses on the same data set were nearly simultaneously reported with our paper in esteemed peer-reviewed journals like Proceedings of the National Academy of Sciences (Berger & Le Mans, 2009), the Proceedings of the Royal Academy (Hahn & Bently, 2003), among other peer-reviewed journals (Bentley, Lipo, Herzog, & Hahn, 2007; Fryer & Levitt, 2004) and naming trends and patterns have been extensively studied and discussed by economists (Steven Levitt and Steven Dubner in the best selling book Freakonomics) and sociologists (Stan Lieberson in A Matter of Taste: How Names, Fashion, and Culture Change). The work we published was peer-reviewed in a journal by scientific experts and went through multiple revisions with extensive debate.

Our paper reports novel findings which suggest a refinement of leading theories of cultural transmission of ideas. The report gets the basic finding from our research wrong when it claims our conclusion was the tautology “popular names are popular with parents.” If only it was so simple. One prediction of the idea that “popular names are popular” would be that the most popular names would never change from year to year (the same popular names would keep being popular). In fact, the historical record provided by the Social Security Administration shows that there has been dramatic changes in the popularity of names over the last few years. Our paper proposes and evaluates possible reasons for these changes in time. Our theory is rigorous and mathematically specified, and may thus be used by other researchers studying the cultural transmission of other ideas (such as political ideologies, health-related habits and decisions, or purchasing decisions). The ideas in the paper borrow from recent mathematical theories of human decision making and learning as well as cultural transmission and cultural evolution.

As authors, we made a concerted effort to communicate the broader impacts of this work to the public at large. The paper is available for free from my website (http://gureckislab.org), and both NYU and Indiana University jointly issued a very nice press release with details and discussion about the merits of the paper which went far beyond the third-party source the Coburn report extensively quotes (the USA Today article).

3. Why publish a paper about naming patterns in the first place? Is this a
useful scientific topic?

It is easy to be distracted by the seemingly trivial nature of “baby naming.” However, as noted above and in the paper, we did not choose this topic for frivolous reasons. Baby naming just happens to be a cultural practice for which there are extensive historical records about the aggregate decisions of millions of individuals. Thus, it provides an important domain in which to test theories of how other people influence our opinions, decisions, and judgements. These theories are far from trivial and involve detailed mathematical arguments about how the distribution of cultural tokens (such as names) should change in response to societal forces. Research should not be singled out simply for pursing theoretically motivated research that just happens to reference a popular culture phenomena. In fact, this feature of this work helps many Americans recognize the potential of NSF funded research to transforming our understanding of the world around us.

Baby naming represents a unique cultural domain in which to test theories of how other people influence our opinions, decisions, and judgements.

While Coburn’s report suggests this research is obvious or trivial, many areas of both public and private research funding are currently very interested in this type of research. For example, understanding the factors that influence how ideas spread through a human groups may help our military better influence the “hearts and minds” of people we are trying to help. In addition, it is noted that social influence has an effect on the health decisions that people make (e.g., Christakis & Fowler, 2007). The application of the ideas in this research may be later used to enact positive societal outcomes. We find evidence that names go through boom and bust cycles not unlike the recent economic bubbles that lead to the current budget situation. Understanding the factors that contribute to these bubbles could be important in preventing these events in the future.

4. Should NSF support for social and behavioral sciences be eliminated?

Coburn’s report recommends that funding for social and behavioral sciences should be terminated within NSF (page 53). Coburn cites the past successes of astronomy, biology, chemistry, and physics as examples of the important research that NSF supports. As someone originally trained as an electrical engineer, I couldn’t agree more that basic research in these areas deserves continued investment. However, investing in basic research based primarily on past success is bad science policy.

Investing in basic research based only on past success is bad science policy.

Many of the future challenges that face our society have to do with human behavior. For example, how can we get people to make better decisions for their health? How does the brain contribute to behavior? How can we best intervene to improve student learning and retention? How can we develop better treatments for language disorders or developmental disabilities? How do trends propagate through society and how might these contribute to “bubble”-like market phenomena? Research funded under NSF’s Social, Behavioral, and Economics Sciences initiative supports ground-breaking research on many of these issues.

Contrary to the impression given in the report, NSF’s SBE includes divisions for “hard” sciences (such as the National Center for Science and Engineering Statistics), decision making, economics, and computational social science (under the division of Social and Economic Sciences), cognitive neuroscience, linguistics, and developmental and learning sciences (under the Behavioral and Cognitive Sciences division), as well as numerous programs focused on improving science education in our country (such as the Research Experiences for Undergraduates program and the Science of Learning Centers). These are not “frivolous” topics but important areas of research that contribute greatly to our economy, our health, our status as a worldwide leader in scientific research, and our national security.

Critically, NSF funds basic research of theoretical importance which has the potential to make truly transformative progress. I would argue that many of the advances in science that we will be talking about in future generations will come from further development of a detailed, quantitative, and mathematically-based science of human behavior, exactly of the type exemplified by the baby naming paper.

Citations:

Bently, R.A., Lipo, C.P., Herzog, H.A., Hahn, M.W. (2007) Regular rates of popular culture change reflect random copying. Evolution and Human Behavior, 28(3), 151-158.

Berger, J., Le Mens, G. (2009), How Adoption Speed Affects the Abandonment of Cultural Tastes, Proceedings of the National Academy of Sciences, 106, 8146-8150.

Fryer, R.G. and Levitt, S.D. (2004) The causes and consequences of distinctively black names. Quarterly Journal of Economics, 119(3), 767-805.

Gureckis, T.M. and Goldstone, R.L. (2009) How You Named Your Child: Understanding The Relationship Between Individual Decision Making and Collective Outcomes. Topics in Cognitive Science, 1 (4), 651-674.

Hahn, M.W. and R.A. Bentley (2003) Drift as a mechanism for cultural change: An example from baby names. Proceedings of the Royal Society London B, Biology Letters, 270:S120-S123.

Christakis, N.A. and Fowler, J.H. (2007) “The Spread of Obesity in a Large Social Network Over 32 Years,” New England Journal of Medicine. 357(4): 370-379.

Drawingmachine by Eske Rex from Core77 on Vimeo.

The artist’s website.

Bottom line: How to create a finite sample of data with a fixed set of summary statistics

Often in psychology experiments we show participants stimuli that are intended to be samples from some distribution. One way of doing this is to simply draw the samples from the distribution (this is easy to do in most computer software packages). For example, we might sample numbers from a Gaussian distribution:

However, in any random finite sample, the mean and other distributional properties will slightly mis-match those of the generating population. A natural response is that, well, it can’t be that far off. For example, we know that mean will vary according to the standard error of the distribution. But with multivariate samples (such as those commonly needed for categorization experiments), the properties of the higher-order statistics, like covariance, are also important. The sampling distribution of these properties can be more distorted by small samples (Klaus Feidler has done some interesting psychological work that exploits exactly this fact).

Ideally, when we say that “we presented subjects in an experiment with items generated from some particular population distribution”, what we really mean is that we showed all participants a random sample of items with properties close or exactly matching to what we are describing about the population distribution. Otherwise, random differences in the sample any subject sees may be contributing to our “unexplained variance” in the experiment design.

How do you guarantee that the statistics of a finite sample match the population exactly? Since the points are already normally distributed, we just need to find a linear transform of the points of the form

to change the mean and variance to the values we want. For univariate Gaussian distributions, the solution is the z-transform:

where the sample mean and variance of are and , respectively. This is written a little different than a classic linear transformation, but basically, we just subtract and divide out the mean and variance that the sample had, giving the sample a mean of 0 and a standard deviation of 1. This is actually the same as taking the z-score for all the numbers in the sample. Once our mean is 0 and standard deviation is 1, we can multiply by any constant to make that scalar our standard deviation, and add by any constant to make that constant our new mean.

This kind of correction can be generalized to multivariate samples using linear algebra.

As before, the plan is to remove the mean and covariance of our original set of points, and then add and multiply in our desired parameters. This process is illustrated in the figure: we start with sample Xold, give it mean of zero and the identity matrix as its covariance (sample Z), then multiply in a new shape and add in a new mean to get our new sample, Xnew. Each of these conversions, represented by the arrows, will be a linear transformation from a given sample, to a new sample with different parameters. Because it is linear, it will take the form

Ashby (1992) has shown that when performing the above transformation, the new covariance matrix can be determined from the old one as follows:

where is the conjugate transpose of . To target a particular value for , it will help to use the Cholesky decomposition. is defined as a triangular matrix such that for any Hermitian positive-definite matrix ,

This is applicable here because a covariance matrix is Hermitian positive-definite. There are other solutions for in this case, but the Cholesky decomposition is particularly efficient and does not require the use of singular value decomposition.

Our goal is to transform our sample into a new sample with and . Once we have that, it will be easy to transform it to have the statistics we want. So we solve the above:

Defining (and therefore ), we get

By the definition of the Cholesky decomposition, , so,

Noting, trivially, that , the transformation to a “clean” sample is now:

which is equivalent to

To transform this into the desired distribution, we again turn to Ashby’s equation relating the transformed and untransformed covariances, but this time the term on the right contains the identity matrix, so it’s even easier:

And so our transformation becomes

Summary:

To fix the sample mean and covariance of a sample from a multivariate Gaussian to arbitrary values, first transform the sample to a sample with mean and covariance .  This can be done using this equation:

Then transform that sample to the sample you want, using

Code:

Some Python code to accomplish this is provided here:

import numpy as NP
# get_cloud - generates a multivariate, finite sample of
# size n, with given mean and cov
def get_cloud(Mu, Cov, n):
    ndims = len( Mu )
    mu0 = NP.zeros(ndims)
    cov0 = NP.eye(ndims)
    zsamples = NP.random.multivariate_normal( mu0, cov0, n ).T
    ecov = NP.cov(zsamples)
    ecovch = NP.linalg.cholesky( ecov )
    ecovchi = NP.linalg.inv( ecovch )
    covzeroed = NP.dot( ecovchi, zsamples )
    Z = NP.add( covzeroed.T, - NP.mean( covzeroed, 1 ) ).T
    covadjusted =  NP.dot( NP.linalg.cholesky( Cov ), Z )
    meanadjusted = NP.add( covadjusted.T, Mu )
    return meanadjusted.T

# example usage - 25 items, mean - [2,2], standard multinormal
data = getcloud([2,3], [[3.,1.],[1.,2.]], 25)
print "Tests (should equal input parameters):"
print "Mean vector:"
print NP.mean( data, 1 )
print "Covariance matrix: "
print NP.cov(data)

References:

Ashby, F.G. (1992). “Multivariate probability distributions.” In F. G. Ashby, Ed., Multidimensional Models of Perception and Cognition. Laurence Erlbaum Associates: Hillsdale, NJ.

Answer: Nothing?

Here’s another TED Talk link (so many good, short ones)! Much of the current work in our lab is devoted to understanding how people learn about the statistical patterns in their environment. One of the core issues we debate is how people figure out the “breaking points” in their input that form different categories of items (be that speech sounds, categories of objects in the world, etc…). For example, how much evidence does one need to decided there are two categories of R/L sounds in your language or just one? The TED talk by Patricia Kuhl linked below provides a really nice developmental perspective on this issue. The experiments she describes show how young children acquire the phonemic distinctions relevant to their language. It’s amazing data and really highlights how our experience shapes how we interpret the world. In addition, work by Kuhl (and many others) is forcing us to rethink the sophistication of young learners (i.e., babies). Here’s a paper from our lab that is particularly relevant to her talk (presenting a computational/mathematical theories of how learners might acquire distinctions, like the japanese/english R/L distinction Kuhl discusses in her work) through learning.

Review: Jones, M., and Love, B. (2011, in press). “Bayesian Fundamentalism or Enlightenment? On the Explanatory Status and Theoretical Contributions of Bayesian Models of Cognition.” Behavioral and Brain Sciences.

In the world of cognitive Psychology, there is a dizzying array of frameworks for building models. For example, to describe a given phenomenon, a researcher could choose to use a “connectionist” or a “Bayesian” model. To an outsider to the field, it might seem these choices are inconsequential: if a theory is ultimately about the nature of human thought, what difference does the mathematical “language” it is expressed with make? Isn’t the more important question to ask if a theory tells us something useful about the mind?

However, as it turns out, the choice of mathematical formalism often does means quite a lot, since it can greatly change what one learns from the model or what the model means.

Over the last couple years, there has been a large movement in the cognitive science community towards developing Bayesian models of cognition. To understand Bayesian probabilistic models and the controversies surrounding their use, it will help to understand a little more about what they are.

Bayesian probabilistic models allow scientists and statisticians to develop models of the “rational” inferences learners should make based on a set of observations, given a mathematically precise description of how the possible states of the world relate to those observations and what the learner’s prior beliefs are about those possible states. These are useful for two reasons:

1. They can be used to solve ordinary statistical problems (e.g., does smoking cause cancer?)
2. They can be used as ideal observer models, answering the question of how humans and animals “should” behave when asked to solve a particular cognitive problem.

Recently, a number of papers have proposed Bayesian models of various aspects of cognition, and given close fits to human data, argued that human behavior is therefore “rational”. This approach has generated outcry from some who feel it encourages a preoccupation with rationality and mathematical formalisms, diverting attention away from the interesting psychological questions of how these problems are solved by human and animal brains.

The central tenet of Bayesian Fundamentalism is the belief that human behavior can be explained entirely through rational analysis, given a correct probabilistic interpretation of the task environment.A recent paper in Behavioral and Brain Sciences [see also] by Matt Jones and Brad Love can be considered a comprehensive manifesto for this viewpoint. Jones and Love critique a perspective which they call “Bayesian Fundamentalism.” The central tenet of Bayesian Fundamentalism is the belief that human behavior can be explained entirely through rational analysis, given a correct probabilistic interpretation of the task environment. Under this view, there is no need to make reference to mechanistic explanations to explain behavior: since humans act rationally, a rational model will fully describe their behavior.

Jones and Love’s primary objections to this paradigm can be summarized as follows:

• Without a careful study of the environment and cognitive challenges that put our ancestors under evolutionary pressure, it is impossible to accurately specify the assumptions that should be built into a model of a cognitive task. Therefore, the predictions of the models are highly unconstrained, and similarity to human behavior cannot be taken as evidence that humans behave rationally.
• Theories of cognition which have no predictions on an algorithmic or implementational level are fundamentally unsatisfying, and that many of the contributions of cognitive modeling to other fields has been in the form of mechanistic predictions.

The authors call for a turn toward “Bayesian Enlightenment,” in which the algorithmic and implementational aspects of probabilistic models are taken seriously as having Psychological content.

We read a version of this paper in our recent lab meeting and our reactions were resolutely mixed. Some felt that the article did an excellent job pointing out theoretical excesses in the field, while others felt that it was overly dismissive of the usefulness of showing how a problem could be incorporated into a Bayesian framework.

One source of frustration that Jones and Love were able to address effectively is the common conclusion among modelers that a good fit to human behavior by a Bayesian probabilistic model indicated that human behavior is in some sense “rational.” As the authors make clear, a model cannot be considered a “rational” account of a cognitive process without a thorough analysis of the natural environment and the cognitive challenges that our brains were evolved to solve (a level of analysis completely missing from recent Bayesian analyses of cognition).

… the spector of the “Bayesian Fundamentalist” is a straw man. Who are these people? … What Bayesian wouldn’t welcome constraining data from neuroscience that supported or could bear directly on their model?

Another important issue they address was the apparent lack of clarity concerning the psychological content of Bayesian models. Since Bayes’ rule itself is trivial, the content of a Bayesian model rests almost entirely in the setting up of the hypothesis space and (often) the choice of an approximation algorithm. Bayesian theorists often attack process-level approaches (such as connectionist models or other process-level accounts) for making a large number of “arbitrary” assumptions. However to the degree that assumptions about priors and the hypothesis space in a Bayesian model are also arbitrary (i.e. not set based on an analysis of the evolutionary environment), then there is no real advantage to either approach. One way to ease this tension is to say that the key psychological contribution of Bayesian probabilistic models is their specification of the hypothesis space, prior, and approximation/optimization algorithm (Jones and Love advocate this approach as “Enlightened Bayes”).

On the other hand, there was a real sense that the best Bayesian modelers, who in fact have greatly contributed to the wide-spread interest in these types of models, are interested in process-level models (e.g., Vul, Daw, Steyvers, Griffiths, Goodman, etc…). What Bayesian wouldn’t welcome neuroscientific data that supported or could bear directly on their model? One real risk from the article is confusing people about what current Bayesian models are actually about, by aligning them with this non-existant bogeyman. Oddly, everyone we could think of who might be a “Bayesian Fundamentalist” had also written compelling papers that Jones and Love would called “Enlightened Bayes”. Is this a paper stirring controversy with no real target?

Ultimately though, a great paper for debate, and will hopefully encourage everyone who works with cognitive models of every kind to think a little bit harder about what their models really mean. It’s also pretty clearly written and might be a great place to start if you are interested in learning more about the value of cognitive models.

Melody Dye also has a fun post up about this article with a lot of colorful quotations.

Citations:

Jones, M., and Love, B. (2011, in press). “Bayesian Fundamentalism or Enlightenment? On the Explanatory Status and Theoretical Contributions of Bayesian Models of Cognition.” Behavioral and Brain Sciences.

Cartoon by Daniele Quercia.

(This article was written with input and ideas from our lab)

Anyway, I posted this video because it contains a concise and impassioned statement about of the importance of theories of behavior. Near the end of his talk there’s a nice quote which is the title of this post: “If you can’t build it, you don’t understand it” (approx.). I think that’s as good of a statement for building formal models of behavior as one can come up with (cognitive, neural, or otherwise). But of course, I’m also originally an engineer! Anyway, check it out here:

However, it might have occurred to you (oh savvy blog reader) that coming up with a big list of where particular thoughts occur in the brain is unlikely to be that helpful. (In fact, this was exactly what phrenologists aimed to do in the early 1800s.) Maps of our thoughts and ideas are not a bad starting place for a rigorous science of mind and behavior, but scientific progress ultimately demands prediction. We don’t want to simple know where ideas occur in the brain, but how they occur, why they occur, and when they will occur. Given the wide-spread attention that thought “localizing” garners in the news, are attempts to localize mental function really the best “psychological science” that we currently have?

A real science of mind?

Burge argues no (and I tend to agree). In particular, the main thesis of his article is that a “real” science of mind has emerged over the last 40 years called “vision science.” Being fortunate to work at one of the premier research institutions in the world for the study of vision (NYU), this comes as no surprise really. Perceptual science (the field which includes vision science but also other forms of sensory perception) has a strong intellectual tradition which values detailed, quantitative approaches to the study of perception. Perhaps if a non-vision scientist were to look at the latest issue of Journal of Vision (aside from noticing the beautiful graphics) they might not recognize much that could be identified as a theory of human behavior.

However, unlike the “neurobabble” (Burge’s term for shoddy neuroscience that makes it into the news), vision scientists aim to develop detailed, testable theories about how people visually perceive their environment. In place of folk psychological theories or even intuitively specified scientific theories, such approaches offer rigorous, mathematically grounded theories that can predict (and explain) behavior across a wide range of situations. The theories specifically try to understand the stages of information processing and representations that people use to interact with the visual world. Since the theories are detailed enough to be expressed mathematically (or as computer programs) there is much less disagreement in the scientific community about what they predict, and it is easier to falsify theories when they are incorrect. It is the kind of rigor which led to important advances in the physical and biological sciences, and is doing the same for the psychological and brain sciences.

What else is there?

Burge’s article made me think about what other areas of behavioral or social science have made progress in developing rigorous testable theories of psychological phenomena. In general, I might argue that the field I work in (cognitive science) meets a similar standard (with a number of extra challenges that vision scientists fortunately don’t have to grapple with), but in fact, I think an even better example is the learning sciences**. The learning sciences may be one of the best examples of a rigorous, mathematical science of behavior that is not primarily perceptual or sensory in nature.
Learning is one of the oldest and most successful areas of psychological science. The pioneering work of Pavlov, Thorndike, Skinner, Hull, Tolman, Rescorla, and others laid the foundation for a rigorous understanding of learning behavior. Much of this early work was done under the (now tarnished) name of Behaviorism, but the ideas still hold considerable influence in the field. Indeed, learning science today focuses on a wide set of issues include how people learn from examples, how they learn to imitate one another, and how they learn to adapt their behavior to changing circumstances.

In fact, I would argue that the learning sciences may be of the best examples of a rigorous, mathematical science of behavior that is not primarily perceptual or sensory in nature. Such work speaks to fundamental questions about human thought (e.g., nature versus nurture), is validated and refined through cross-species comparisons, and is backed up with in detailed, quantitative modeling. There is even a growing understanding of the neural mechanisms supporting learning (including high quality fMRI research finding parts of the brain responsible for adapting behavior). What’s particularly interesting about the learning sciences is that they have a strong potential for immediate impact on a number of important societal issues (e.g, How best do we structure learning environments to help children learn? What role does nature or nurture play in learning? How can we build machines that can learn as well as people do? What is the role of learning in addiction?).

Anyway, I’ve always admired the precise and careful control that vision scientists adopt in their stimuli, tasks, and theories. However, relative to flashier imaging findings, this work is often overlooked (particularly in media reports about psychological science). That’s why it is great to have someone like Burge highlighting this work in a high-profile outlet. I guess I’d just like to add the learning sciences as another example of a “real” science of mind.

** My definition of “learning sciences” is considerably more broad that the intuition given here which focuses on cognitive-educational theories. My view is that these areas form a contiguity with the more general psychological study of learning and memory (from multiple perspectives including neuroscience, cross species work, etc…) and with machine learning/artificial intelligence. It’s a big tent, but as all the features of a emerging, quantitative, and socially relevant science. A good summary was published by Meltzoff et al. in Science recently.

One of my favorites is the clip with my former advisor and collaborator Rob Goldstone about how the mind (or maybe more accurately “information”) is a different kind of stuff than studied in the physical sciences. Another great one is Jeff Elman talking about the relationship between the mind, brain, and computation. Check out the full list here.

Why us? We are a research group working to learn about human cognition and to pursue research in this area (pretty sweet gig, eh?). While there are a large number of excellent blogs devoted to cognitive science topics we hope that our contributions can be a useful resource for students, teachers, researchers, and the general public looking to learn more about the cognitive sciences (i.e., artificial intelligence/machine learning/psychology/linguistics/developmental psychology/neuroscience/etc). What makes our blog different from all these great existing online resources perhaps remains to be seen, but we have it in mind to provide accessible discussions cutting-edge peer-reviewed research, technical notes that we develop in the course of our research which may help others, random bits of interesting CogSci miscellany, general musings, and lab news with an eye to the relevance of cognitive science to everyday life. The additional advantage (for us at least) is that it forces us to digest, engage, and keep on top of the work in our area!

Who are we? “We” is the Computation and Cognition lab at New York University under the direction of Todd M. Gureckis. The “we” is grad students and researchers, undergraduate students, and other guests who plan to contribute stories and articles for the lab.

What do we do? Our lab studies human cognition (in particular human learning). The name of our lab is “computation and cognition” which reflect the two main strands that inform our work. Principally, we attempt to understand the nature of human cognition by developing computational theories of human information processing. What is a computational theory? Well, basically, it means we believe that the human mind can be productively understood as a information processing device similar to your laptop or computer. The question then is what is the “software” that the brain runs (i.e., what algorithms do people’s brains use to solve particular tasks/problems?). In our research, we propose and test various computational algorithms as candidate theories of how people think. By comparing the operation formal models and the behavior of humans, we can gain precise, scientific insight into how people perceive their environment, solve problems, and adapt their behavior. In addition, insights from these models can inform the development of artificial systems capable of learning on their own (such as in machine learning or artificial intelligence systems). You can read more about our lab here (a collection of peer-reviewed research papers are here).

What next? Thanks for tuning in, and please stick around and join us for some good old fashioned internet debates and discussions about Cognitive Science!