Inspiring people to work together to build a more vibrant and stronger community.

follow: @arlemjm

Monday, November 10, 2014

How Google Is Using Dynamic Testing To Build Out The Knowledge Graph


Google is constantly testing new ways to answer user queries. Columnist Eric Enge has been watching these experiments closely and shares what he's observed.

Google’s Knowledge Graph is dynamically changing. We suspected as much as we saw changes during the Knowledge Box Showdown I published earlier this month.
For that reason, I personally redid 250 of the 3,086 queries we used in the study, to see what would happen. The results were interesting to say the least.
Continuous Testing Leads to Continuous Improvement
The TL;DR is that Google appears to be testing various possible direct answers to search queries. If the results are not to its liking, it is trying a different answer source, until user data tells them they have one that works.
In some cases, it decides it does not have a good enough answer and stops trying to show a knowledge box (aka answer box) altogether.


In this test, I took 250 queries that we used in our tests, 125 which previously returned no answer box, and 125 that did. In addition, of the 125 that did, I picked only the ones that returned step-by-step instructions. Each of these was retested, and I checked for:
  1. Did it return step-by-step instructions?
  2. Did it return a knowledge box other than step-by-step instructions?
  3. If yes, to 1 or 2, did it use the same website as a source as the first time we tested them?
For the ones that previously returned no answer box, if Google does now, I noted if the new result was in the form of step-by-step instructions, or some other type of knowledge box.
As in the original study, all of the searches were performed using voice queries on a phone. In this case, I used the Google App running on an iPhone 4.

The Findings

For the 125 queries I tested that originally returned answer boxes, here is how it shook out:
Changes in Step by Step Results
I found that 75% of the tested items still show a step-by-step instruction result. The rest of these changed. In addition, the sites used to respond to the queries also changed. Here is how that broke out:
  1. Of the 94 cases that still returned step-by-step results, 15 of them (16%) now show results from a different site.
  2. Of the 16 that now show knowledge panels, 10 of them (62.5%) show results from a different site.
For the 125 queries I tested that originally showed no answer box, here is what we saw:
Changes in Queries that Returned no Answer Boxes
Across the 250 total tested items, we can tabulate the following changes:
  1. 20 fewer step-by-step instructions.
  2. 3 total more answer boxes (23 new non step-by-step answer boxes).
For purposes of this tabulation, the image results were treated as “no knowledge box” even though they did attempt to answer the question with those responses.


My speculation is that Google is dynamically testing various answer sources and specific ways of answering questions, and seeing what works based on user data. If it likes the results, it sticks with it; if it doesn’t, it is trying the next possible answer. When it has no satisfactory answer, it falls back to the web search results.
Its algorithms can’t determine the perfect answer, so it calculates what it considers best possible matches, and then tests it in the real world laboratory of search, making you and me the QA team for how good the results are.
If the best possible match does not work, it may try the next best possible match, and so forth. It can also run tests in parallel by showing different results from different data centers to speed this process.
While this is aggressive on Google’s part, but the company’s belief in the concept of answer boxes of all kinds is evidently quite strong, and this shows Google is willing to do what it takes to be able to show good answers.
© j.a.Molina Creative | All rights reserved.
Blogger Template Designed by pipdig