Data Mining and Predicting Human Behavior

Interesting article up at NYTimes on using data mining to predict societal outcomes. Much of the article is devoted to the ethical implications of these data-mining efforts. However, I still have doubts about how useful this stuff actually is. As the article notes, social-media based prediction is often less accurate than more traditional means of gathering information. Perhaps some of the fear about these systems has to do with how they are named (e.g., “Total information awareness”). It’s easy to get confused on what these kind of things can actually do. Predicting “everything” isn’t really how systems like this can practically work. A better, more concrete example is the Google flu trends system which asks a specific question: “do flu symptom searches predict outbreaks?” It’s always helpful to know what the specific question is before thinking about if data mining is the answer!