New test shows that Google Assistant is the smartest virtual personal assistant

Virtual personal assistants were tested in a new study that contained 4,952 questions that were asked to all four assistants. Each response was divided into one of five categories. Did the assistant answer verbally? Was the answer sourced from a database like the Knowledge Graph? Did the answer come from a third party source like Wikipedia? Did the assistant fail to understand the question? Did the assistant understand the question, but fail to answer correctly?

The four virtual personal assistants tested were Alexa, Microsoft's Cortana on the Invoke smart speaker, Google Assistant on the Google Home, Google Assistant on a smartphone, and Siri. While all four can handle other requests such as setting an alarm, ordering flowers and getting dinner reservations, this testing was done strictly to measure which assistant is the most knowledgeable.

The overall winner was Google Assistant on a smartphone. The latter attempted to answer the most questions (in other words, it said things like "Sorry, I don’t know that" fewer times than the competition) and had the highest percentage of "fully and correctly answered"questions. Cortana on the the Invoke was second in the percentage of correct answers, followed by Google Assistant on Google Home, Alexa and then Siri. Back in February, Apple's virtual assistant finished last in a study that was limited to assistants from smart speakers only.

Comparing the 2018 test to last year's results, only Cortana on the Invoke showed an improvement in the percentage of correct answers (86% to 92.1%). Google Assistant on a smartphone was not part of the study last year. While Alexa's percentage declined, it did increase the number of questions it attempted to answer by 2.7 times over last year. Cortana had the lowest percentage of flat out incorrect answers, followed by Google Assistant on both a smartphone and on a Home smart speaker. All of those assistants were wrong well under 1% of the time. On the other hand, the percentage of wrong answers spoken by Alexa rose sharply from just under 1% in 2017 to over 3% this year. Siri answered incorrectly about 3% of the time last year, and a bit over 3% this year.

Google Assistant on a smartphone had the largest number of answers that were attributed to a third party source. This happened on over half of its responses. Alexa had the fewest number of answers sourced from a third-party, while the Invoke's Cortana showed the only year-to-year decline in this category. While Siri relied on a third party source only a bit over 20% of the time, it had the largest year-to-year increase. Alexa had the largest percentage of questions that it answered with a joke, followed by Siri, which had a big decline in this category.

This test seems to confirm what has become obvious; Siri is losing ground quickly. The test also tells us that perhaps Alexa has become a bit overrated and Cortana underrated. It also reveals that Google Assistant is actually the smartest virtual personal assistant at this point in time.


source: StoneTemple