Hi,

I am interested in using a chatbot for customer support and am trying to figure out how to evaluate their performance. I am looking for a way to test different chatbots in regards to the ability to understand enquiries and match them with answers.
I know that e.g. Pandorabots provide this as a basic function, but one still has to provide a lot of permutations of each potential enquiry for a specific topic/answer. Other systems like cleverscript seem to have pattern recognitions to reduce the effort, but I wanted to see what the experienced members of this group recommend.

The turing test isn’t that important for what I am looking for as that seems to be focused on the “human likeness”. I am more interested in how to get the question - answer pairs as quickly up an running and with a high accuracy + flexibility ( I know I ask for a lot). I always hoped a great NLP engine would understand “like a human” and make that job easier.

Thanks for the help.
Bora

If you currently have a customer support system in place, taking the log of the transactions or the FAQs from a web site is a good place to start.

Most of the chatbots use some form of pattern matching for their “NLP Engines”. This is how many of the automated customer support systems were created.

Thanks Merlin. Any suggestions on how to test them / evaluate them against each other? I wouldn’t want to build up on a system and then find out there is another one much better suited.

The key is building your set of test questions. Get a list together that encompasses the types of queries you expect. That is probably the most important element.

Almost any of the systems might do the job if you put in enough effort. You can ask for opinions on this board.
There are a few things to keep in mind:
How will the system be deployed (self hosting)?
Commercial or home grown?
Amount of on-going support required?

The types of people who may handle the programmatic set-up and maintenance of a bot/AI interpreter may be different than the editor/maintainer of the knowledge base (different skill sets required).

I had hoped somebody might have put together already an interesting sample of test questions to do such a test

You can see in the thread on types of questions, the typical queries available.
http://www.chatbots.org/ai_zone/viewthread/1370/

I haven’t come across a list that is specifically focused on customer service questions.