By Meagan Clark -
An economics professor has found a way to estimate U.S. employment from Twitter data, more quickly and accurately than the government.
University of Michigan Professor Matthew Shapiro and his UM colleagues searched tweets for words and phrases people commonly use to talk about jobs and unemployment, like “lost work.” Then Shapiro, Margaret Levenstein of the Survey Research Center and computer scientists Michael Cafarella and Dolan Antenucci tested the results to make sure the terms are referring to jobs and not, in the case of “lost work,” a computer crashing.
“When we started,” Shapiro told UM, “we had no idea if we could track job loss with tweets, but over a two-year period, we’ve seen the social media index perform quite well.”
In 2011 to 2012, their social media index leveled off, reflected fluctuations around Hurricane Sandy and the government shutdown in late 2013, in line with official government data. When the social media index didn’t match government data, it turned out to be more accurate, like when California got new computers but there were delays in processing unemployment claims that caused government data to underestimate employment, Shapiro said.
Many economic indicators like the government’s estimated employment rate and the university’s well-known consumer sentiment index, are based on mail and phone surveys. But these surveys are costly and becoming difficult as fewer people have landline phones or respond to surveys at all. The data when released is often four to six weeks old. Mining Twitter, on the other hand, provides data in real time.
“In turning points or times of crisis, having something available instantaneously is critical for policy makers,” Shapiro said.