Test Automation to deliver Virtual Assistants

Challenges faced while testing virtual assistants

A user can interact with virtual assistants in multiple ways. The conversations may begin and end in different contexts. For e.g. a user may ask a virtual assistant about the location of a nearby coffee shop. The virtual assistant may show him the directions to a nearby Starbucks store. Immediately the user starts enquiring about Starbucks and the conversation would start building in that direction. So, what started as an enquiry towards finding a nearby coffee shop ended in enquiring about a brand. This shows that the way a user would interact with virtual assistants cannot be contained under a category. Similarly, the virtual assistants’ responses are also not static. If a user says “Hi” to a virtual assistant at 5 instances, the response may vary each time. The virtual assistant may respond as “Hi”, “Hello”, “Good day”, “Good Evening”, etc. This shows that the virtual assistant responses are dynamic in nature. Most of the virtual assistants would be built on Artificial Intelligence (AI), Machine Learning (ML) or Neuro Linguistic Programming (NLP) based engines. There would also be a huge volume of data, in the region of millions, 
present behind each virtual assistant. Hence, the conventional methods of testing would not be sufficient to validate virtual assistants. We would need to shift towards AI based testing techniques to validate virtual assistants. As per Forrester, 63% of customers will leave within 2 minutes of just one poor experience related to delays in processing responses by virtual assistants. Hence, the testing shouldn’t be confined to only the functional aspects rather the virtual assistants need to be evaluated on non-functional parameters too.

Read More