In search of the best, general purpose algorithm for counterbalancing conditions
Some updates based on discussion thread
We are looking for input on the best general purpose counterbalancing algorithm for experiment designs. We’ve identified some of the features that we believe such an algorithm should have as well as part of the way to do it, but we’re interested if we are missing some general purpose solution that already exists out there. Please comment if you have thought about this!
Here are the desired features:
- Subjects are pseudo-randomly assigned to conditions.
- Conditions fill in more or less evenly. For example, if you have four conditions in your experiment and run 20 people there should be 5 in each condition. If you run 200 there should be 50 in each condition. Another way of saying this is that if you take any reasonably sized sub-sequence of subjects (e.g., subject 23-33 in a design with 100 people and four conditions) the conditions will still be roughly balanced (basically the process has approximate ergodicity).
- When there are multiple nested conditions (e.g., N-way designs, or counterbalancing of random factors like button placement), these should also fill in evenly but with lower priority than the main experimental conditions. For example, imagine a study where people are assigned to two experimental conditions like shape (triangle or circle) but are assigned to one of 25 different colors. You care mostly about shape in your experiment but want color randomly and evenly assigned as well. If you run 20 people you want 10 to see triangles and 10 to see circles. For the color you don’t have enough subjects to visit all 50 combinations. However, you want color to be assigned evenly in your sample as well (so you don’t have 2-3 people with blue triangles).
- You may want to enforce conditional dependencies between factors. For example, you want the distribution over assignments to color to be separately balanced for both the circle and triangle group, or you are just choosing color independently of the subjects assignment to a shape. Choosing independently can lead to issues with correlation: it may be that every time you have a blue shape, it’s always a triangle, which could make your data hard to analyze. However, if you choose conditionally (i.e. choose the color randomly given that it’s a triangle), it could be that you end up with many more blue objects than red objects overall in your experiment. There is an obvious tradeoff here and you may want to optimize for your particular experiment.
You want all this to work in a situation where people might withdraw from your study at any time (e.g., online studies). This is the kicker. In a normal psychology study, you just assign subjects equally to every possible combination of conditions and don’t worry about it. If someone drops out, you just assign the next person who comes in to the same condition (or someone else later on). But often in online experimentation, we have 30 people taking an experiment at the same time and we are unsure which of them are actually going to finish (in other words, which conditions are actually going to end up in your analysis). You need to assign people to conditions in a way that’s robust to dropouts so that in the end the distribution of conditions is pretty even. Thus you need to be able to adjust your assignment algorithm given some reasonable idea about who is unlikely to return a data point.
Is there a literature on this? Has this been worked out by someone? It seems like the above features cover quite a broad range of experiment designs. We suppose people often optimize these types of balancing by hand after deciding on a sample size for an experiment, however an algorithmic solution might be useful. We got inspired about this a bit by the recent Facebook PlanOut library, although that is designed for different types of experiments than most psychology experimenters deal with.
So far we created a simple, open source Python library called Counterbalancing which accomplishes a few of these objectives. If anyone is interested in contributing to this to make it even better, please jump in!