I’ve been reading snippets of Eli Pariser’s book, The Filter Bubble: What the Internet Is Hiding From You.
He leads off the book with a discussion of the effect of Google’s “personalization” feature on the ranking of search results. This feature uses 54 signals (what browser version you’re using, your prior searches, geographic location, and so on) to customize search results for each user.
Pariser was concerned about this and tested it by asking two friends to run the same search at the same time and comparing the results. He found that the results were disturbingly different, and concluded that search engines are
“increasingly biased to share our own views. More and more, your computer monitor is a kind of one-way mirror, reflecting your own interests while algorithmic observers watch what you click.”
I was intrigued but skeptical; this sounded like PR-driven hyperbole. Yeah, Google does some customization that you can’t turn off, but is it really that bad? Is Google really filtering my news?
I decided to test this by asking a group of fellow info-entrepreneurs to conduct a search for the word Israel in Google News and send me a screen shot of their search results page. I received a total of 37 responses. [If you want to know about the methodology and other nitty-gritty details, scroll down to the bottom of this post.]
What I found surprised me; there was more variation among search results than I had expected. Knowing that a graphic is worth 1000 bytes (or something like that), here are a few ways of telling the story.
Caveat: I was a philosophy major in college and never had to take a stats course, so this analysis was done without the benefit of any relevant skills. I’m happy to go through the details of my analysis with you if you’re interested, or if you would like to share your thoughts on other ways to display this information.
What it means: Each of the black vertical bars represents a story that appeared in the search results. The red bars represent the percentage of searchers who did not see that story. As the chart indicates, one story appeared in more than 90% of the search results, another appeared in 70%, and then the bottom drops out. Of the remaining 14 stories, none were seen by more than 30% of the searchers, and most were seen by less than 15% of the searchers.
What it means: Only 12% of searchers saw the same three stories in the same order; 88% saw something different or the stories were in different order. Pariser was right; we really do see different versions of the news.
What it means: More than a quarter of the stories showed up in only one searcher’s search results; if I want to conduct a thorough search, I know I’m going to have to dig deeper into the search results.
What it means: Almost one in five searchers saw a story that no one else saw. Again, this confirms Pariser’s charge that there’s a lot of filtering going on behind the scenes.
Bottom line: Holy moley, Google does filter the news. You really need to go beyond the first few search results if you want to get a relatively well-rounded view of the news.
Methodology and other details
I made my query on August 22 on the private list of the Association of Independent Information Professionals (aiip.org) and received 37 responses within six hours. While it is fairly common knowledge, at least among info pros, that Google search results vary widely from one searcher to another, I had assumed that I would see far less variation in Google News searching.
Another assumption was that the group I was polling are people who need to find the most useful information for a project and would have minimized any behind-the-scenes filtering. Since I was asking for a quick favor, I made the request as simple as possible: “Just take a screen shot of what you see on the screen and send it to me.” That meant that most of the responses only displayed the first three or four headlines. For consistency, I included only the first three results of each response.
I threw all the data into a spreadsheet, capturing info on what titles appeared, in what order, in how many countries, and by how many individuals. Since there were 26 responses from the US and no other country was represented by more than four people, most of my analysis was on the US responses, checking with the aggregate data to see whether there were any large discrepancies.
I treated all stories about a specific issue as equivalent, even if the specific headlines in the search result differed. So, for example, I counted as equivalent “Three Gaza rockets fired into Israel, breaking ceasefire” and “Gaza militants renew rocket fire despite truce”.