AI Zone Admin Forum Add your forum

NEWS: Chatbots.org survey on 3000 US and UK consumers shows it is time for chatbot integration in customer service!read more..

How would you detect a chatbot?
 
 

To the experts here who compete to create chatbots that can act like people, let me ask: what would you do to detect a chatbot? How would you avoid being fooled?

Are the Russian political “bots” detectable? How would you police Twitter?

Is there a single kind of question that would trap all but the most sophisticated bots?

 

 
  [ # 1 ]

The main thing to realise is that there is a great diversity of bots, but each is limited in domain, so your best tactic would be to try very varied questions.

The average chatbot or spambot won’t be able to handle deliberately abusive misspelling, but a commercial chatbot will, because they can afford the resources involved in spellchecking. This is still the most effective tactic though.

Second-best is asking word games, like “How many vowels in the xth word of your last response?”. Because there is no reasonable purpose for word games and there are many variations possible, few bots are made to answer them.

However, with chatbots I generally just look at how often they return generic all-purpose replies like “yes, I do” (do what?), or if they keep referring to what I said as “it” for several bouts without topic-relevant details. A lot of bots don’t handle pronouns because that requires resource-heavy parsing.

As for troll bots, whenever I found insinuative vile spewing posts they typically turned out to be from real Americans. Bots’ subject matter seemed more focused on controverse than provocation. The following features are common:
- Outlandishly foreign-seeming usernames with awkward and long combinations of syllables (generated)
- Repetitive subjects and frequent posting history (only equalled by some obsessive autistics)
- Lack of interactive activities (no replies, no following, followers, interests or subscriptions)
- Bad grammar (similar to Indian and Russian, with verbs and prepositions out of order)
- Bare-bones profile information is a bonus
For the time being, troll bots are best detectable through meta-data and posting history. Since a bot’s purpose is to spam, it should post as often as it can, without for instance the inescapable 8-hour period of inactivity known as sleep. I heard there’s a famous Russian bot in America that starts anger-Tweeting at 3AM in the morning every day. That’s definitely not human.

 

 
  [ # 2 ]

A wonderful list. Thanks.

I was thinking not too many bots can answer simple questions that combine generalization, arithmetic, and pronouns:
“I have 3 apples and Jon has 4 oranges: how many pieces of fruit does he have?”

 

 
  [ # 3 ]

You’ll also find some interesting suggestions here: https://www.chatbots.org/ai_zone/viewthread/1776/

I’m afraid the questions that one intuitively considers difficult, often turn out to be easily sidestepped. Mostly because everybody else also thinks they are good questions, so chatbots will have encountered them before. Fruit arithmetic has been so regular in the Loebner Prize that one can do with recognising two numbers and “how many” and ignore everything else for a pretty good guess. Of course then you think, “What if I ask how many elephants?” and you’d probably find that that also occurred to the chatbot creators.
But, I should mention that this is only true for chatbots and most particularly Turing Test participants. It is highly unlikely that spambots or commercial bots would be equipped for that line of questioning, because it is not relevant to their domain.

 

 
  login or register to react