Is Google really filtering my news?

I’ve been reading snippets of Eli Pariser’s book, The Filter Bubble: What the Internet Is Hiding From You.

He leads off the book with a discussion of the effect of Google’s “personalization” feature on the ranking of search results. This feature uses 54 signals (what browser version you’re using, your prior searches, geographic location, and so on) to customize search results for each user.

Pariser was concerned about this and tested it by asking two friends to run the same search at the same time and comparing the results. He found that the results were disturbingly different, and concluded that search engines are

“increasingly biased to share our own views. More and more, your computer monitor is a kind of one-way mirror, reflecting your own interests while algorithmic observers watch what you click.”

I was intrigued but skeptical; this sounded like PR-driven hyperbole. Yeah, Google does some customization that you can’t turn off, but is it really that bad? Is Google really filtering my news?

I decided to test this by asking a group of fellow info-entrepreneurs to conduct a search for the word Israel in Google News and send me a screen shot of their search results page. I received a total of 37 responses. [If you want to know about the methodology and other nitty-gritty details, scroll down to the bottom of this post.]

What I found surprised me; there was more variation among search results than I had expected. Knowing that a graphic is worth 1000 bytes (or something like that), here are a few ways of telling the story.

Caveat: I was a philosophy major in college and never had to take a stats course, so this analysis was done without the benefit of any relevant skills. I’m happy to go through the details of my analysis with you if you’re interested, or if you would like to share your thoughts on other ways to display this information.

 

What it means: Each of the black vertical bars represents a story that appeared in the search results. The red bars represent the percentage of searchers who did not see that story. As the chart indicates, one story appeared in more than 90% of the search results, another appeared in 70%, and then the bottom drops out. Of the remaining 14 stories, none were seen by more than 30% of the searchers, and most were seen by less than 15% of the searchers.

What it means: Only 12% of searchers saw the same three stories in the same order; 88% saw something different or the stories were in different order. Pariser was right; we really do see different versions of the news.

 

 

 

What it means: More than a quarter of the stories showed up in only one searcher’s search results; if I want to conduct a thorough search, I know I’m going to have to dig deeper into the search results.

 

 

 

What it means: Almost one in five searchers saw a story that no one else saw. Again, this confirms Pariser’s charge that there’s a lot of filtering going on behind the scenes.

 

 

Bottom line: Holy moley, Google does filter the news. You really need to go beyond the first few search results if you want to get a relatively well-rounded view of the news.

Methodology and other details

I made my query on August 22 on the private list of the Association of Independent Information Professionals (aiip.org) and received 37 responses within six hours. While it is fairly common knowledge, at least among info pros, that Google search results vary widely from one searcher to another, I had assumed that I would see far less variation in Google News searching.

Another assumption was that the group I was polling are people who need to find the most useful information for a project and would have minimized any behind-the-scenes filtering. Since I was asking for a quick favor, I made the request as simple as possible: “Just take a screen shot of what you see on the screen and send it to me.” That meant that most of the responses only displayed the first three or four headlines. For consistency, I included only the first three results of each response.

I threw all the data into a spreadsheet, capturing info on what titles appeared, in what order, in how many countries, and by how many individuals. Since there were 26 responses from the US and no other country was represented by more than four people, most of my analysis was on the US responses, checking with the aggregate data to see whether there were any large discrepancies.

I treated all stories about a specific issue as equivalent, even if the specific headlines in the search result differed. So, for example, I counted as equivalent “Three Gaza rockets fired into Israel, breaking ceasefire” and “Gaza militants renew rocket fire despite truce”.

 

Comments

[Richard Torian]

Isn’t Google News a feed – meaning that new news stories might continuously come down the feed as news comes from various sources?

If yes, then I am not surprised that different stories (feeds) would be seen over a period of time. And this could account for the variety that was observed.

Richard Torian

[Mary Ellen Bates]

Yes, Google News is a feed, but that's not what this survey looked at. Remember that this was a test of *search*, not of what happens when you just go to the Google News front page and see the news feed.

I chose a topic (Israel) that is reliably in the news but not the subject of a breaking news story that day -- a test that I thought would address your concern about variation over the course of several hours. And I assumed that, whatever filters may have accumulated on searchers' browsers, if it affected a search like this, it probably affected all their "real" searches too.

When I tested this ahead of time, searching for the word Israel in Google News over the course of a day, I found that I reliably saw the same stories in the same order. The exact headline varied; I accounted for that in my analysis, by combining all headlines that covered the same story, regardless of which one was featured.

[Dennis O'Connor]

Try using chrome's incognito browser window (or any of the other browser's anonymous options) as a starting place. Then cross check search results, news results etc.

This approach should strip out personalization and yield a more realistic picture of what the Google Index holds.

Of course if everyone is using the 'personalized approach' and our concern is what people see or find when they search, we are out of luck.

Dennis O'Connor (21st Century Information Fluency 21cif.com)

[Lt. Uhura]

The flipside of this concern is an appreciation that we all are NOT getting the exact same results/feed/brain-narrowing. There is NO PERFECT SEARCH methodology. Any will have weaknesses and prejudices. In the past it was more heavily weighted on page views/popularity rankings/links. SEO became the name of the game.
Google's search engine/method may result in a more natural result -- that is more diversified and organic based on who is searching and who they are evolving to be (as seen through their online activity/interests).
Google may be scary & cause for concern, but not uniquely so and not without some possible benefits to a diverse society.